[JZ]jelleorange [JJ]jaapgreen [JS]jakubred [LB]lisablue

# Black Big Boxes: Do Language Models Hide a Theory of Adjective Order?

Jaap Jumelet1 Lisa Bylinina2 Willem Zuidema1 Jakub Szymanik3
1ILLC 2Center for Language and Cognition 3CIMeC University of Amsterdam
University of Groningen University of Trento
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In English and other languages, multiple adjectives in a complex noun phrase
show intricate ordering patterns that have been a target of much linguistic
theory. These patterns offer an opportunity to assess the ability of language
models (LMs) to learn subtle rules of language involving factors that cross
the traditional divisions of syntax, semantics, and pragmatics. We review
existing hypotheses designed to explain Adjective Order Preferences (AOPs) in
humans and develop a setup to study AOPs in LMs: we present a reusable corpus
of adjective pairs and define AOP measures for LMs. With these tools, we study
a series of LMs across intermediate checkpoints during training. We find that
all models’ predictions are much closer to human AOPs than predictions
generated by factors identified in theoretical linguistics. At the same time,
we demonstrate that the observed AOPs in LMs are strongly correlated with the
frequency of the adjective pairs in the training data and report limited
generalization to unseen combinations. This highlights the difficulty in
establishing the link between LM performance and linguistic theory. We
therefore conclude with a road map for future studies our results set the
stage for, and a discussion of key questions about the nature of knowledge in
LMs and their ability to generalize beyond the training sets.111All code and
data, including CAP, can be found here: github.com/jumelet/lm-adjorder

##  1 Introduction

Figure 1:  We connect the adjective order preferences (AOP-$\Delta$, §3.1) of
language models (here Pythia-12b) to the adjective order frequencies of the
corpus they have been trained on (The Pile). We highlight various regions of
interest: adjective pairs for which both orders are rare and that require the
LM to generalize from other adjective orders; pairs for which one particular
order is far more common that can be resolved from frequency alone; and orders
with high variance that rely more strongly on context.

Linguistic capabilities of language models (LMs) have been subject to a lot of
research and benchmarking (e.g., Marvin and Linzen 2018; Hu et al. 2020;
Gauthier et al. 2020). While LMs show impressive fluency in language
generation and often score high on linguistic benchmarks, the mechanisms
underlying their linguistic proficiency are hard to uncover. To what extent do
LMs generalize beyond mere memorisation of training data? Do they acquire
abstract linguistic rules and constraints that don’t boil down to basic co-
occurrence patterns? How similar are linguistic generalizations in LMs to
those that humans have?

These questions have been asked and partially answered for different
constructions and grammatical phenomena. The focus of such studies has mostly
been on strict and binary grammatical constraints, where a violation results
in an ungrammatical sequence.

There has been less attention to how ‘softer’ linguistic constraints are
captured by LMs. Such constraints can give rise to slight and defeasible
preferences rather than sharp acceptability contrasts. One example is
adjective order in noun phrases where multiple adjectives modify one noun,
e.g. large wooden box. Changing the order of adjectives does not necessarily
lead to unacceptability, but can result in a slight decrease in naturalness
(Dyer et al., 2023) (‘>’ marks preference):

  1. large wooden box $>$ wooden large box

Moreover, these preferences are context-dependent: often, a context can be
found where preferences switch – for instance, when a contrasting property is
introduced (Teodorescu, 2006).

  1. Take a wooden large box, not a plastic one.

  2. $>$ … large wooden box …

These adjective order preferences (AOPs) have been subject to a lot of
interest in theoretical linguistics, but proved difficult to capture due to
their graded and context-sensitive nature (see Levshina et al. 2023 for
arguments in favor of a systematically gradient approach to word order). A
number of factors ranging from superficial properties like adjective length to
semantic and pragmatic properties to information-theoretic properties of a
sentence have been identified as predictors for AOP. But even with all those
factors combined together, a large portion of AOPs remains unaccounted for:
the best model combining different predictors explains only 71% of human AOPs
(Dyer et al., 2023).

Adjective order offers an opportunity to assess the ability of LMs to learn
subtle linguistic preferences involving implicit, hard-to-identify factors.
This paper is the first study of AOP in LMs. We put together metrics and data,
and conduct a series of experiments that explore AOP in LMs, its relation to
properties of training data and the extent to which LMs learn more abstract
principles behind AOP, generalizing to unseen items. The contributions of our
paper are the following:

  * •

We introduce a reusable Corpus of Adjective Pairs (CAP*X) and a series of
metrics to measure AOP in LMs (§3).

  * •

We show that LMs are impressively good predictors of naturally occurring
adjective orders, with AOP prediction accuracy as high as 94.1%. We identify
three stages of AOP acquisition in LMs that lead to their performance at the
end of the training (§4).

  * •

We demonstrate that linguistic and cognitive predictors only partly account
for LM performance in AOP prediction. We identify the gap between linguistic
theory and LM performance in the AOP domain that needs to be explained (§5).

  * •

To pinpoint the role of training data co-occurrence statistics in LM AOP
predictions, we study the Pile (Gao et al., 2020) – a large-scale corpus that
Pythia models were trained on. Simple bigram statistics turn out to be a
powerful AOP predictor for naturally occurring adjective orders on their own,
at 90.3% accuracy (§6). Figure 1 summarizes a close relation between count
statistics and LM predictions, questioning whether LM predictions are
significantly driven by anything beyond $n$-gram counts.

  * •

At the same time, we find two ways in which LMs go beyond simple collocational
statistics: 1) We test LM predictions for adjectival combinations not seen
during training (a challenging setup due to the large size of the Pile; we
overcome this obstacle by turning to intermediate checkpoints and
corresponding portions of training data). We find some—but
limited—generalization to new data (§6); 2) We study the effect of context on
AOP. We conclude that the sentence helps LMs make better AOP predictions
compared to noun phrase with adjectives taken in isolation or in a set of
random contexts, which indicates that AOP mechanisms in LMs go beyond $n$-gram
counts in training data (§7).

##  2 Background

###  2.1 Adjective Order Theory

The mechanisms that drive adjective order have been a topic of research for
decades (see Dyer et al. 2023 for overview). Theories have been proposed as
far back as the 19th century (Sweet, 1898), and refinements to the theory are
being made to this day. Early theories proposed lexical hierarchies that
described adjective order based on abstract classes such as dimension or color
(Dixon, 1982; Sproat and Shih, 1991; Cinque, 1996; Scott, 2002). Adjectives
were expected to be ordered based on the class they belong to: e.g. dimension
goes in front of color, which gives rise to the order of a phrase like the big
red house.

While these hierarchies are powerful predictors of adjective order, they are
not really explanatory: the question remains how such hierarchies are formed
by the underlying properties of adjectives. Furthermore, corpus studies have
shown that many counterexamples against various versions of these hierarchies
can be found (Truswell, 2009), demonstrating the need for a more fine-grained
analysis (Trotzke and Wittenberg, 2019).

One line of work looks for semantic driving forces of AOP, placing emphasis on
adjectival properties like intersectivity and subsectivity (Truswell, 2009) or
identifying conditions under which adjective order constraints can be lifted,
such as focus (Vendler, 1968; Teodorescu, 2006).

Other approaches take a broader perspective, leveraging insights from
quantitative linguistics, distributional semantics and information theory, and
connecting this to psycholinguistic theories of human language processing.
Early analyses focus on properties such as length—shorter adjectives tend to
precede longer ones (Behaghel, 1930)—and frequency: more frequent adjectives
tend to appear earlier (Martin, 1969; Ney, 1981; Wulff, 2003). This follows
from a broader theory of language production, according to which more
accessible and familiar material is produced first (Bock, 1982; Ferreira and
Dell, 2000) and is comprehended faster (Arnon and Snider, 2010). Collocational
patterns of adjectives and nouns have been shown to be highly predictive of
adjective order: adjectives most closely connected to the noun in term of co-
occurrence appear closer to it (Sweet, 1898; Byrne, 1979; Lapata et al.,
1999). This intuition has been operationalized through pointwise mutual
information (PMI) (Hahn et al., 2018; Futrell et al., 2020a), which aligns
with more general theories of word order arguing that distance between words
that closely predict each other tends to be small (Gibson, 1998; Futrell et
al., 2020b).

Predictors themselves can be complex, based on collocational or semantic
factors or combinations thereof: modification strength of adjectives, for
instance, is a predictor based on adjectival compositional potential (Vecchi
et al., 2013, 2017); other predictors proposed in the literature include
information gain that an adjective provides for a noun (Dyer et al., 2021),
and subjectivity of adjectives, with more subjective adjectives occurring
first (Hill, 2012; Scontras et al., 2017, 2019; Franke et al., 2019). Although
the majority of these approaches focus on English, various cross-lingual
accounts have demonstrated their universal nature: adjective order tends to be
driven by the same principles across languages (Wulff and Gries, 2015; Leung
et al., 2020; Scontras, 2023; Dyer et al., 2023).

Finally, various papers highlight the need for a multi-factorial account of
adjective order: it is unlikely adjective order is driven by a single factor
and therefore it’s the result of various competing pressures (Wulff, 2003;
Trotzke and Wittenberg, 2019; Dyer et al., 2023). Relatedly, a recent position
paper (Levshina et al., 2023) argues for a fundamentally gradient theory of
word order that would be able to capture its hybrid and multi-faceted nature.

In our work, we compare LM predictions to various adjective order theories.
Additionally, we examine the utility of LMs for investigating the role that
context plays on word order: since almost all theories focus on adjectives in
isolation, LMs may provide a fruitful starting point for shaping a contextual
account of adjective order.

###  2.2 Word Order in Language Models

With the increasing capabilities of language models, their linguistic
abilities have been under much scrutiny in recent years. A wide range of
procedures have been introduced to uncover their understanding of
grammaticality and notions of linguistic structure (Linzen and Baroni, 2021).
Grammaticality is often assessed using the LM’s probability predictions of
minimal pairs of grammatical and ungrammatical sentences (Marvin and Linzen,
2018; Warstadt et al., 2020; Gauthier et al., 2020); phenomena that have been
examined this way include number agreement (Linzen et al., 2016; Gulordava et
al., 2018), negative polarity items (Jumelet and Hupkes, 2018; Bylinina and
Tikhonov, 2022), and filler-gap dependencies (Wilcox et al., 2018;
Suijkerbuijk et al., 2023). To our knowledge, no prior work has focused on
adjective order in particular: the closest related phenomenon to this would be
Misra and Mahowald (2024)’s analysis of adjective-numeral constructions in
LMs.

The importance of word order in LMs has been a topic of debate, with various
works claiming that downstream performance is not affected by scrambled inputs
(Malkin et al., 2021; Sinha et al., 2021), although it has been shown that LMs
are able to retain a notion of word order through their positional embeddings
(Abdou et al., 2022). It has been argued that LMs acquire an abstract notion
of word order that goes beyond mere $n$-gram co-occurrence statistics (Futrell
and Levy, 2019; Kuribayashi et al., 2020; Merrill et al., 2024), a claim that
we in this paper assess for large-scale LMs in the context of adjective order.
Finally, numerous works have investigated the trade-off between memorization
and generalization in LMs: it has been shown that larger LMs are able to
memorize entire passages from the training data (Biderman et al., 2023a; Lesci
et al., 2024; Prashanth et al., 2024), but generalization patterns for
grammatical phenomena have also been shown to follow human-like generalization
(Dankers et al., 2021; Hupkes et al., 2023; Alhama et al., 2023).

##  3 Methods

###  3.1 Measuring Word Order Preference

We measure a language model’s AOP by comparing its log probability of the
natural order (the order in which a pair of adjectives appears in the sentence
from our corpus) with a swapped order. In our experiments we measure AOP in
two settings, either by considering the phrase in isolation222We compute these
isolated probabilities by prepending the noun phrase with ‘The’., or by taking
the context of the phrase into account. For a double adjective phrase
$\textsc{a}_{1}\textsc{a}_{2}\textsc{n}$ that is extracted from a corpus with
context c, we compute a model’s preferred order as follows:

$\displaystyle\textup{AOP-}\Delta(\textsc{a}_{1}\textsc{a}_{2}\textsc{n})$
$\displaystyle=P(\textsc{a}_{1}\textsc{a}_{2}\textsc{n})-P(\textsc{a}_{2}\textsc{a}_{1}\textsc{n})$
(1)
$\displaystyle\textup{AOP-}\Delta(\textsc{a}_{1}\textsc{a}_{2}\textsc{n}|\textsc{c})$
$\displaystyle=P(\textsc{a}_{1}\textsc{a}_{2}\textsc{n}|\textsc{c})-P(\textsc{a}_{2}\textsc{a}_{1}\textsc{n}|\textsc{c})$
(2)

All probabilities are in log space. The AOP-$\Delta$ metric represents the
magnitude of a models’ preference. In our experiments we also consider an
accuracy metric based on AOP-$\Delta$, expressed as the number of items for
which AOP-$\Delta$ is positive:

$\displaystyle\textup{AOP}\textit{-\%}(\mathcal{C})=\frac{\left\\{\phi\in\mathcal{C}\mid\textup{AOP-}\Delta(\phi)>0\right\\}}{|\mathcal{C}|}$
(3)

Where $\mathcal{C}$ is an evaluation corpus of double adjective phrases, as
described in the following section. Misra and Mahowald (2024) consider a
similar metric for measuring properties of word order preferences in LMs,
based on corrupted word orders.

###  3.2 CAP*X: Corpus of Adjective Pairs

To evaluate adjective order preferences across a wide range of adjective pairs
and contexts, we extract a novel set of double adjective phrases from a
corpus. Earlier work has extracted adjective pairs from the Universal
Dependencies treebank (Dyer et al., 2023), but in order to collect a larger
range of adjective pairs we collect novel constructions ourselves. We release
this evaluation corpus under the name of CAP*X: Corpus of Adjective Pairs.

#### Procedure

As a starting point, we use the BabyLM 10M corpus (Warstadt et al., 2023),
which consists of a mix of various English language sources including
Wikipedia articles, books, and transcribed dialogue data. From this corpus we
make a pre-selection of sentences containing multiple adjectives by filtering
all sentences containing at least two adjectives, based on a list of 11,000
adjectives released by Futrell et al. (2020a). Next, we generate dependency
parse trees for each sentence using spaCy and select all sentences in which a
noun contains two amod relations to an adj that is part of the adjective list.
This procedure results in 2678 double adjective sentences. We provide a sample
of CAP sentences in Appendix A.1.

From these sentences we also generated corrupted sentences in which the
adjective order is swapped. In case the first adjective is preceded by an
indefinite article we ensure the article is changed accordingly to satisfy
morpho-phonological constraints. Here is an example of an extracted phrase and
its swapped counterpart:

  1. $+$

Peter received a full athletic scholarship [..]

  2. $-$

Peter received an athletic full scholarship [..]

###  3.3 Models

For our experiments we make use of the Pythia suite of language models
(Biderman et al., 2023b). The models are released in increasing sizes (from
70M to 12B parameters) and all trained on The Pile corpus (Gao et al., 2020).
Importantly, all intermediate checkpoints are available for these models in
logarithmic intervals, which allows us to investigate the learning dynamics at
a fine-grained level.

Figure 2: A–B: AOP-% and AOP-$\Delta$ scores for Pythia models of increasing
size. C–D: AOP-% and AOP-$\Delta$ scores for Pythia-1.4b during training. We
highlight the three learning phases: 1) initialization, 2) acquisition, and 3)
consolidation.

##  4 Experiment 1: AOP in LLMs

In this first set of experiments we evaluate a series of pretrained LLMs on
CAP using the AOP-$\Delta$ and AOP-% metrics of §3.1.

#### Model Size

For the eight Pythia models of increasing size we plot the AOP-% performance
in Figure 2A, and the AOP-$\Delta$ performance in Figure 2B. It can be seen
that performance improves with model size: contextual AOP-% performance
increases from 89.6% for the 70m model to 94.1% for the 12b model. AOP-% does
not increase consistently with size, however, from the 1b model onward it
reaches a plateau. Interestingly, AOP-$\Delta$ does increase consistently with
size: this shows that larger models become more certain about their AOP
judgment.

#### Learning Dynamics

To investigate how AOP develops during training, we compute scores for the
1.4b model on all its intermediate checkpoints in Figure 2C and 2D. From these
plots we can identify three distinct learning phases: 1) an initialization
phase in which AOP judgments have not been formed yet, 2) an acquisition phase
in which AOP judgments are rapidly being formed, and 3) a consolidation phase
in which AOP judgments remain stable and only the role of context is being
reinforced. A similar three-phase learning dynamic has been observed by Van
Der Wal et al. (2022) and Chen et al. (2023). Note that the consolidation
phase is reached relatively quickly: after around 2000 batches, which is ~1.4%
of the total model training. This demonstrates that adjective order is a
relatively low-level linguistic feature that is acquired early during
training. Furthermore, it highlights the importance of investigating learning
dynamics at a logarithmic scale: with linear checkpoints we would not obtain
this fine-grained level of precision.

Figure 3: Average token probabilities for the original and swapped adjective
orders on Pythia-12b, without and with sentence context (A–B), as well as the
token-level differences (C) that correspond to the difference between the
curves in (A) and (B).

#### Localizing AOP

In our experiments so far we have measured AOP based on the probabilities of
the full adjective-noun triplet. Preference order can arise at different
locations of the phrase. The most natural location for this is the probability
of the second adjective after having seen the first adjective, but context
could also play a role for preferring which adjective comes at the first
position, and a noun could be more probable after having seen two adjectives
in their natural order.

In Figure 3 we plot the average token probabilities on CAP for Pythia-12b, in
isolation and contextual. We observe that, as expected, the largest
probability difference occurs at the position of the second adjective: the
model is highly surprised by encountering the second swapped adjective after
having seen the first. The noun probability is strongly affected by swapping
adjective order as well: a noun is more likely to follow adjectives in natural
order. Furthermore, we can see that context plays a role for the probability
of the first adjective already: whereas in the isolated case both adjectives
in first position receive equal probability, order preference already
manifests here when the model has access to context.

#### Conclusion

The Pythia LMs acquire a strong sense of adjective order that is learned early
in training at around 1.5% of the total amount of training. We showed that
sentence context has a positive effect on AOP, bumping up AOP-% from 91% to
around 94%. By localizing the AOP scores on the token-level we show how these
order preferences manifest at different points in a phrase. In the next
experiments we will investigate these results in more detail, connecting LM
AOP to cognitive predictors and training data statistics.

##  5 Experiment 2: Cognitive Predictors

Metric | AOP-% | $\rho(\textup{AOP-}\Delta(\bullet))$ | $\rho(\textup{AOP-}\Delta(\bullet|c))$
---|---|---|---
AOP-$\Delta(\bullet)$ | 91.6 | – | 0.71
AOP-$\Delta(\bullet|c)$ | 94.1 | 0.71 | –
[1pt/1pt] a | 49.6 | 0.16 | 0.10
PMI(a;n) | 62.4 | 0.27 | 0.20
Subj(a) | 69.3 | 0.07 | 0.08
[1pt/1pt] $\%(\textsc{a}_{1})$ | 60.5 | 0.21 | 0.18
$\%(\textsc{a}_{1}\textsc{a}_{2})$ | 90.3 | 0.58 | 0.53
$\%(\textsc{a}_{1}\textsc{a}_{2}\textsc{n})$ | 87.6 | 0.54 | 0.53

Table 1: Predictive accuracy of adjective order on CAP for various metrics:
AOP-$\Delta$ for Pythia-12b, 3 cognitive predictors (§5), and relative n-gram
counts from The Pile (§6). The highest accuracy for each group is bolded. For
each metric we provide the Spearman correlations with respect to the
AOP-$\Delta$ scores of Pythia-12b.

As discussed in §2.1, a range of cognitively plausible predictors have been
proposed to explain adjective order. In the previous section we have shown
that LMs are highly capable of predicting adjective order, and in this section
we investigate how their predictions are related to various theories on
adjective order. We evaluate the performance of three cognitive
predictors—Length, PMI, and Subjectivity— and correlate them against
AOP-$\Delta$ scores across training.

###  5.1 Cognitive Predictors

For each predictor we compute its score for a particular adjective pair. An
adjective order predictor $f$ returns a graded score for adjective order, akin
to our formulation of AOP-$\Delta$.

#### Length

For Length—which hypothesizes that the first adjective is the shortest—the
score for adjectives a1 and a2 is expressed as

$f(\textsc{a}_{1},\textsc{a}_{2})=|\textsc{a}_{2}|-|\textsc{a}_{1}|$

We also considered expressing length as the number of tokenizer subwords,
since this is the only notion of length LMs have access to, but found that
this led to similar results as string length.

#### PMI

We follow the procedure of Futrell et al. (2020a) for computing PMI, which is
based on the distribution over adjective relations in dependency parse trees.
It is defined as

$\textup{PMI}(\textsc{a};\textsc{n})=P(\textsc{a},\textsc{n})/(P(\textsc{a})\cdot
P(\textsc{n}))$

$P(\textsc{a},\textsc{n})$ is computed as the normalized number of occurrences
of adjective a and noun n in amod relation. $P(\textsc{a})$ and
$P(\textsc{n})$ are obtained by marginalizing over this joint distribution.
PMI hypothesizes that adjectives closer to the noun have a higher PMI:

$f(\textsc{a}_{1},\textsc{a}_{2},\textsc{n})=\textup{PMI}(\textsc{a}_{2};\textsc{n})-\textup{PMI}(\textsc{a}_{1};\textsc{n})$

#### Subjectivity

We leverage the 450 subjectivity ratings provided by Dyer et al. (2023), to
compute subjectivity scores for the subset of adjective pairs that contain a
rating. This procedure retains 37.5% of the adjective pairs in CAP (1005
items). Subjectivity as predictor hypothesizes that more subjective adjectives
occur first (Scontras et al., 2017):

$f(\textsc{a}_{1},\textsc{a}_{2})=\textup{Subj}(\textsc{a}_{1})-\textup{Subj}(\textsc{a}_{2})$

###  5.2 Predictive Accuracy

We present the predictive accuracy of each predictor in Table 1. Subjectivity
is the strongest predictor with 69.3% accuracy, followed by PMI (62.4%) and
Length (49.6%). While Subjectivity reaches the highest accuracy, its
correlation with the LM’s AOP-$\Delta$ scores is surprisingly the lowest of
the three. We plot these correlations over time in Appendix A.2: all three
correlations develop in the acquisition phase and remain stable in the
consolidation phase.

We also compute the overlap between these three predictors on the 1005 items
for which we have subjectivity scores:

The first Venn diagram makes clear that the cognitive predictors have low
agreement: only 213 orders are predicted correctly by all three. The broad
coverage (93.1%) of the union, however, shows that most of the CAP items are
explained by at least one of these predictors. In the four diagrams on the
right we show the overlap of the predictors with the AOP-$\Delta$ scores of
Pythia-12b. All three predictors and their intersection are almost fully
subsumed by the AOP-$\Delta$ predictions. This demonstrates that
AOP-$\Delta$’s unique contribution goes significantly above that of the
ensemble of the cognitive factors, raising the question whether LMs ‘hide’ a
meaningful linguistic theory that extends beyond established factors.

##  6 Experiment 3: Is AOP Driven by Data Statistics?

Figure 4: Correlations during training of LM probabilities for single
adjectives, adjective pairs, and adjective-noun triplets with respect to their
frequency in The Pile.

In order to truly understand what LMs base their adjective order preferences
on, we need to get a better insight into how double adjective constructions
are distributed in the training data. The way that contemporary LMs are
trained nowadays makes this challenging: many models are released without open
access to their training data, and the scale of training corpora that are
available poses a challenge for targeted analysis.

###  6.1 WIMBD

To overcome this, we make use of the WIMBD API (What’s In My Big Data, Elazar
et al., 2023), which provides a fully indexed search engine over various
large-scale corpora such as The Pile, on which the Pythia models have been
trained. WIMBD makes it possible to retrieve the exact $n$-gram statistics
that LMs have been trained on, and as such provides a detailed insight into
the collocational factors that play a direct role in shaping AOP.

We retrieve the uni-, bi- and trigram counts for all the adjective-noun
triplets in CAP for The Pile, as well as for the swapped orders. To provide an
analogue to AOP-$\Delta$, we express the n-gram counts as log count
differences. We express these relative n-gram counts as

$\%(\textsc{a}_{1}\textsc{a}_{2}\textsc{n})=\log\\#(\textsc{a}_{1}\textsc{a}_{2}\textsc{n})-\log\\#(\textsc{a}_{2}\textsc{a}_{1}\textsc{n})$

with similar formulations for adjective bigrams
(%($\textsc{a}_{1}\textsc{a}_{2}$)) and unigrams (%($\textsc{a}_{1}$)).

#### Predictive Accuracy

We present the predictive accuracy of these relative n-gram occurrences in
Table 1. %(a1) predicts adjective order with 60.5%, which provides weak
evidence that more frequent adjectives tend to occur first (Martin, 1969;
Trotzke and Wittenberg, 2019). %(a1a2) is the strongest predictor of adjective
order—except for the LMs—at 90.3%. This shows that a large majority of the
items in CAP can be resolved based on bigram statistics alone, which is
leveraged by the LMs during training. However, n-gram collocations alone do
not fully explain LM performance: a large amount of trigrams are not part of
the Pile and performance for $\%(\textsc{a}_{1}\textsc{a}_{2}\textsc{n})$
drops to 87.6%.

#### Correlations

To investigate how n-gram statistics play a role in the formation of
AOP-$\Delta$ during training, we compute the Spearman correlation between
various log-transformed n-gram counts and a LM’s AOP-$\Delta$ on intermediate
checkpoints. We plot this in Figure 4, for contextual and isolated
AOP-$\Delta$. The trends we observe here align closely with the three learning
phases that we outlined in §4. The initialization phase (1) is marked by a
rapid acquisition of unigram and bigram statistics, that remain unaffected by
context at this point. Then, at the start of the acquisition phase (2), in
which AOP performance rapidly improves, the model starts to incorporate
linguistic context in its uni- and bigram predictions. Finally, in the
consolidation phase (3), trigram predictions start being affected by context
as well, whereas unigram and bigram predictions remain relatively stable. The
fully trained model strongly reflects these correlations to data statistics,
which explains the strong relation we observed in Figure 1 between bigram
frequencies and AOP-$\Delta$.

###  6.2 Generalizing to Unseen Adjective Orders

Figure 5:  The contextual AOP-% performance for Pythia-70m and 1.4b across
training, split out for items that have been seen 0, 1, 2 to 10, and more than
10 times at each specific checkpoint.

Having access to the n-gram distribution a model was trained on allows us to
investigate its performance on unseen adjective orders: is the model able to
generalize from an abstract signal of adjective order to novel orders?
However, due to the enormous size of The Pile (300B tokens) it turns out that
number of unseen adjective pairs in CAP is too small to draw significant
conclusions from (only 4 out 2678 pairs). This leaves us with two options:
either we collect an additional sample of rare adjective pairs that are not
part of the Pile, or we investigate the model at intermediate checkpoints
where a greater amount of adjective pairs has not been encountered yet. An
issue with collecting highly rare pairs is that their canonical order will be
much harder to determine, for which we would require extensive human judgments
on the most natural order. We therefore go for the second option, and set up a
procedure to measure the data statistics during training.

#### Intermediate $n$-gram Counts

Since WIMBD only provides $n$-gram statistics for The Pile as a whole, we need
to collect the intermediate statistics ourselves. Using the Pythia batch
viewer (Biderman et al., 2023b) this becomes possible: our implementation is
able to process the first 10% of The Pile (up to batch 14,000; 29.4B tokens)
in around 3 hours. We collect the bi-gram counts of all adjective pairs in CAP
on the batch level, allowing us to determine exactly when and how often a
specific adjective pair has been seen by the model.

#### Unseen Adjective Pairs

Based on these intermediate $n$-gram counts, we compute the AOP-% scores for
various splits based on the number of times an adjective pair has been
encountered. We focus on the cases where the swapped order has never been
encountered: this allows us to isolate the moment a model encounters an
adjective combination for the first time, and measure its ‘zero-shot’
performance on cases that are not already skewed towards to swapped order. For
each model checkpoint we collect the $n$-gram counts at that particular point
in training, and split the CAP items based on the number of times an a1a2 bi-
gram has been seen. We provide the size of these splits across training in
Appendix A.3.

We present the results for Pythia-70m and 1.4b in Figure 5. The performance on
unseen pairs is remarkably high, reaching around 85% at the highest point for
both models. Interestingly, the gap between unseen pairs and pairs seen once
is far greater for the 1.4b model than the small 70m model. This shows that
this larger model is able to acquire adjective order preference faster: a
single exposition to an order immediately leads to an increase in
AOP-$\Delta$. We provide a more detailed item-level analysis for these items
in Appendix A.4: while it is clear on the split level that a single occurrence
improves AOP-%, we do not observe such a clear effect on the item level.
Interestingly, as the number of unseen adjective pairs decreases, the
performance on unseen pairs decreases as well. This shows that the predicting
the order for the set of unseen pairs around batch 2,000 (6% of CAP) could be
better inferred from a general signal of adjective order than the set at batch
14,000 (1.3% of CAP).

#### Conclusions

The question we set out to answer in this set of experiments is whether AOPs
are driven by data statistics. We approached this question both in relation to
the $n$-gram distribution of the Pile and to the shifting $n$-gram
distribution across training. On the one hand, AOP is strongly driven by data
statistics: we find a strong correlation between AOP-$\Delta$ and count-based
AOP scores, which is also visualized in Figure 1. However, the relatively
strong performance on the set of unseen adjective pairs during training
indicates that LMs are able to infer order from a more general signal for
adjective pairs it has not encountered before.

##  7 Experiment 4: The Role of Context

Figure 6:  A: Item-level AOP-$\Delta$ scores with and without sentence-context
for Pythia-12b. We provide a sample of sentences from each quadrant in
Appendix A.1. B: The relative impact ($c$-$\Delta$, Eq. 5) of an item’s true
context compared to an expectation over random contexts (Eq. 4).

In §4 we established that LMs have a thorough grasp of adjective order that
manifests itself early during training. We showed that having access to
sentence context boosts performance, and we now investigate the role of
context in more detail.

#### Contextual Improvements

To gain a better insight how much items are impacted by context, we
investigate AOP-$\Delta$ scores on an item-level with and without context. In
Figure 6A, we plot the AOP-$\Delta$ scores in isolation against the contextual
scores, and mark the four quadrants of items that result in a positive or
negative score after adding context. In the majority of cases (89.4%),
AOP-$\Delta$ is positive both with and without context: the model is able to
determine adjective order based on the adjective-noun triplet alone. In 4.8%
of the cases, however, adding context flips a negative AOP-$\Delta$ score to a
positive one. Adding context has a boosting effect in general: in 65.1% of the
cases, AOP-$\Delta$ is increased by adding context. We provide an overview of
how these ratios are affected by model size in Appendix A.5: the number of
cases where context improves AOP-% is actually larger for small models, but so
is the number of cases where context worsens it.

#### Random Contexts

The fact that context improves AOP-$\Delta$ raises the question what
information in the context is responsible for this improvement. Does context
contain relevant semantic information that plays a meaningful role for
determining adjective order, or is any linguistic signal sufficient for this?
To test this, we compute an expectation over AOP-$\Delta$ scores with random
contexts, in which an adjective pair is preceded by a randomly sampled context
from another adjective pair in CAP:

$\textup{AOP-}\Delta(\textsc{a}_{1}\textsc{a}_{2}\textsc{n}\mid\mathcal{C})=\mathbb{E}_{c\sim\mathcal{C}}\left[\textup{AOP-}\Delta(\textsc{a}_{1}\textsc{a}_{2}\textsc{n}\mid
c)\right]$ (4)

We compute the expectation over a random sample of 500 contexts, taken from
CAP ($\mathcal{C}$). To express the relative impact of adding context, we
regress out a model’s isolated AOP-$\Delta$ score by subtracting it from the
contextual scores, which we call $c\textup{-}\Delta$:

$c\textup{-}\Delta(\bullet|c)=\textup{AOP-}\Delta(\bullet|c)-\textup{AOP-}\Delta(\bullet)$
(5)

By computing this score for both the original context and for the expectation
over random contexts we can quantify how much more important the original
context is for AOP-$\Delta$.

We plot the results for this experiment in Figure 6B. The true context
improves isolated AOP-$\Delta$ in 65.1% of the cases, whereas random contexts
lead to an improvement for only 39.6%. Furthermore, for 33.2% the true context
leads to an improvement whereas random contexts don’t: this subset can serve
as a valuable starting point for investigating what properties of the context
are important for adjective order.

#### Conclusions

AOP in Pythia is context-sensitive. Providing the model with the sentence
where the adjective combination appears increases the model’s preference
towards the target adjective order – both compared to the noun phrase in
isolation and to the noun phrase in a random context. This result points in
the direction of a more linguistically interesting mechanism for AOP than
reliance on $n$-gram counts in training data alone. In particular, it shows a
non-trivial interaction between the noun phrase and bigger context that turns
out relevant for adjective placement. Establishing this opens up a possibility
of studying the forms and types of this contextual interaction – a potential
novel contribution to linguistic theories of adjective order.

We leave a systematic discovery of types of AOP context-dependence open for
future work, and focus on a small-scale qualitative study for now. An
inspection of context-sensitive cases from CAP reveals at least two ways in
which context matters for AOP:

#### 1\. Structural parallelism

The ‘canonical’ word order can be overridden when context contains a parallel
noun phrase with adjectives in non-canonical order (the example shows
contextual AOP according to Pythia-1.4b):

  * a)

The green big cars AOP-$\Delta$ 4.2

  * b)

The story is not about small red houses, it’s about big green cars
AOP-$\Delta$ 6.63

#### 2\. Property saliency

The property salient in the context brings the adjective describing this
property to the left edge of the noun phrase. If the sentence is about color,
color adjective will come first; if the sentence is about size, it will bring
size adjectives to the left (again, orders preferred by Pythia-1.4b):

  * a)

They asked my favourite color of houses and I said I liked red small houses
AOP-$\Delta$ 0.32

  * b)

They asked my favourite size of houses and I said I liked small red houses
AOP-$\Delta$ 1.61

These observations, of course, barely scratch the surface of possible
contextual AOP interactions, but exemplify the power of LMs in incorporating
contextual information into word order judgments.

##  8 Discussion

#### Abstract Notions of Adjective Order

In this paper, we looked for AOP generalization in LMs. We were interested in
whether abstract driving forces are at play in LM predictions beyond simple
$n$-gram counts in training data. From our experiments we conclude that LMs
show limited AOP generalization. Here, we want to highlight an important
question that is rarely brought up in linguistic generalization discussions in
NLP: How general and abstract is human linguistic behavior?

Linguistic literature suggests that human language production, including word
order preferences, is guided by a mix of memorization and abstract principles
(O’Donnell, 2015). Speakers tend to recycle more routinized and therefore more
accessible combinations (MacDonald, 2013), guided by sequence frequencies in
linguistic input that, in turn, form sequential associations that lead to
production automatizing (Bybee, 2010; Diessel, 2019). This is compatible with
a picture of language acquisition where knowledge of grammar is formed based
on encountered exemplars of usage, which are stored in memory (Bybee, 2010)
and serve as basis of generalization (Goldberg, 2006), but can also serve as
patterns for routinized language production. So, human linguistic knowledge
and language production is not totally abstract and is not based exclusively
on generalization. This has been discussed across linguistic phenomena and has
been suggested for word order (MacDonald, 2013; Levshina et al., 2023),
although not for adjective order specifically.

When investigating generalization in LMs it is therefore important to make a
distinction between the degree to which they create generalizations for a
particular phenomenon, and how similar this generalization is to human
behavior. In principle, one could even find cases of LM overgeneralization, if
we are interested in how human-like LMs are linguistically. Importantly, in
order to answer the question of human-like generalization, we would need data
on the trade-off in memorization and generalization in human linguistic
behavior. Such data does not exist yet for adjective order preferences.

#### LMs and Linguistic Theory

Despite the impressive performance of LMs in recent years, their impact on
theoretical linguistics has remained minimal (Baroni, 2022). There are various
reasons for this. On the one hand, the cognitive plausibility of current LMs
remains questionable, both in terms of the scale at which they are trained
(Huebner et al., 2021; Warstadt et al., 2023) and their inductive biases that
depend strongly on model architecture and are not reflective of human language
processing (Wilson and Frank, 2023; Oh et al., 2024). On the other hand, their
black box nature makes it highly challenging to extract linguistic patterns
that can be tested against existing linguistic theories. While advancements in
interpretability have made this latter point less problematic, the number of
works explicitly connecting LM behavior to human language processing remains
few and limited to simple syntactic phenomena (e.g., Lakretz et al., 2021).

In this paper, we explore the possibility that LMs may ‘hide’ a theory of
adjective order, by encoding features that are not covered by linguistic
theories on adjective order. We show that LMs far exceed the predictive power
of prior cognitive predictors, which raises the question whether this increase
stems from meaningful interpretable features. While a lot of the LM
predictions can be explained by collocational patterns—and those statistics
alone do not explain the distribution of adjective orders—we do find some
generalization to unseen adjective combinations and a linguistically
meaningful pattern in the way that context boosts AOP. We hope our work can
serve as an inspiration for using LMs as tools for measuring context-
dependence, which is hard to quantify with traditional quantitative methods.

#### Corpus Interventions

Our analysis of AOP abstractness is constrained by the fixed training corpus
the models are trained on. More detailed insights into this question can be
obtained by a intervention-based approach, in which we filter out specific
constructions from the training data (Jumelet et al., 2021; Misra and
Mahowald, 2024; Patil et al., 2024). An interesting experiment in this
direction would be to remove all sentences containing double adjectives from
the training corpus, and measure whether the remaining distribution of
adjectives contains a sufficient signal to acquire adjective order
constraints.

## Limitations

Our analysis only focuses on a single language model family (Pythia), in a a
single language (English). To test the cross-lingual properties of adjective
order, a wider range of language models should be examined. Our double
adjective extraction method (§3.2) depends on the quality of the dependency
parser that was used, and as such contains inaccuracies. Furthermore, a
downside of using such a broad-coverage approach is that it becomes harder to
draw targeted conclusions about the linguistic properties of cases on which
the model fails or succeeds. Annotating cases containing properties that have
been shown to be important for adjective order in the linguistics literature
would provide deeper insights into model behavior on a linguistic level.

## References

  * Abdou et al. (2022) Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022.  Word order does matter and shuffled language models know it.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6907–6919, Dublin, Ireland. Association for Computational Linguistics.
  * Alhama et al. (2023) Raquel G. Alhama, Ruthe Foushee, Daniel Byrne, Allyson Ettinger, Susan Goldin-Meadow, and Afra Alishahi. 2023.  Linguistic productivity: the case of determiners in English.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 330–343, Nusa Dua, Bali. Association for Computational Linguistics.
  * Arnon and Snider (2010) Inbal Arnon and Neal Snider. 2010.  More than words: Frequency effects for multi-word phrases.  _Journal of Memory and Language_ , 62(1):67–82.
  * Baroni (2022) Marco Baroni. 2022.  On the proper role of linguistically-oriented deep net analysis in linguistic theorizing.  In _Algebraic systems and the representation of linguistic knowledge_ , pages 5–22. CRC Press.
  * Behaghel (1930) Otto Behaghel. 1930.  Von deutscher Wortstellung.  _Zeitschrift für Deutschkunde_ , 44:81–89.
  * Biderman et al. (2023a) Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023a.  Emergent and predictable memorization in large language models.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 28072–28090. Curran Associates, Inc.
  * Biderman et al. (2023b) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023b.  Pythia: A suite for analyzing large language models across training and scaling.
  * Bock (1982) Kathryn Bock. 1982.  Toward a cognitive psychology of syntax: Information processing contributions to sentence formulation.  _Psychological Review_ , 89:1–47.
  * Bybee (2010) Joan Bybee. 2010.  _Language, usage and cognition_.  Cambridge University Press.
  * Bylinina and Tikhonov (2022) Lisa Bylinina and Alexey Tikhonov. 2022.  Transformers in the loop: Polarity in neural models of language.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6601–6610, Dublin, Ireland. Association for Computational Linguistics.
  * Byrne (1979) Brian Byrne. 1979.  Rules of prenominal adjective order and the interpretation of “incompatible” adjective pairs.  _Journal of Verbal Learning and Verbal Behavior_ , 18(1):73–78.
  * Chen et al. (2023) Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. 2023.  Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms.  _CoRR_ , abs/2309.07311.
  * Cinque (1996) Guglielmo Cinque. 1996.  _On the evidence for partial N-movement in the Romance DP_ , Cambridge Studies in Linguistics. Cambridge University Press.
  * Dankers et al. (2021) Verna Dankers, Anna Langedijk, Kate McCurdy, Adina Williams, and Dieuwke Hupkes. 2021.  Generalising to German plural noun classes, from the perspective of a recurrent neural network.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 94–108, Online. Association for Computational Linguistics.
  * Diessel (2019) Holger Diessel. 2019.  _The grammar network: How linguistic structure is shaped by language use._ Cambridge University Press.
  * Dixon (1982) Robert M. W. Dixon. 1982.  _‘Where Have All the Adjectives Gone?’ and other Essays in Semantics and Syntax_.  Mouton de Gruyter, Berlin.
  * Dyer et al. (2021) William Dyer, Richard Futrell, Zoey Liu, and Greg Scontras. 2021.  Predicting cross-linguistic adjective order with information gain.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 957–967, Online. Association for Computational Linguistics.
  * Dyer et al. (2023) William Dyer, Charles Torres, Gregory Scontras, and Richard Futrell. 2023.  Evaluating a century of progress on the cognitive science of adjective ordering.  _Transactions of the Association for Computational Linguistics_ , 11:1185–1200.
  * Elazar et al. (2023) Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. 2023.  What’s in my big data? _CoRR_ , abs/2310.20707.
  * Ferreira and Dell (2000) Victor S. Ferreira and Gary S. Dell. 2000.  Effect of ambiguity and lexical availability on syntactic and lexical production.  _Cognitive Psychology_ , 40(4):296–340.
  * Franke et al. (2019) Michael Franke, Gregory Scontras, and Mihael Simonic. 2019.  Subjectivity-based adjective ordering maximizes communicative success.  In _CogSci_ , pages 344–350.
  * Futrell et al. (2020a) Richard Futrell, William Dyer, and Greg Scontras. 2020a.  What determines the order of adjectives in English? comparing efficiency-based theories using dependency treebanks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2003–2012, Online. Association for Computational Linguistics.
  * Futrell and Levy (2019) Richard Futrell and Roger P. Levy. 2019.  Do RNNs learn human-like abstract word order preferences? In _Proceedings of the Society for Computation in Linguistics (SCiL) 2019_ , pages 50–59.
  * Futrell et al. (2020b) Richard Futrell, Roger Philip Levy, and Edward Gibson. 2020b.  Dependency locality as an explanatory principle for word order.  _Language_ , 96:371 – 412.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The pile: An 800gb dataset of diverse text for language modeling.
  * Gauthier et al. (2020) Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020.  SyntaxGym: An online platform for targeted evaluation of language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 70–76, Online. Association for Computational Linguistics.
  * Gibson (1998) Edward Gibson. 1998.  Linguistic complexity: locality of syntactic dependencies.  _Cognition_ , 68(1):1–76.
  * Goldberg (2006) A.E. Goldberg. 2006.  _Constructions at Work: The Nature of Generalization in Language_.  Oxford linguistics. Oxford University Press.
  * Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hahn et al. (2018) Michael Hahn, Judith Degen, Noah D. Goodman, Dan Jurafsky, and Richard Futrell. 2018.  An information-theoretic explanation of adjective ordering preferences.  _Cognitive Science_.
  * Hill (2012) Felix Hill. 2012.  Beauty before age? applying subjectivity to automatic English adjective ordering.  In _Proceedings of the NAACL HLT 2012 Student Research Workshop_ , pages 11–16, Montréal, Canada. Association for Computational Linguistics.
  * Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020.  A systematic assessment of syntactic generalization in neural language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1725–1744.
  * Huebner et al. (2021) Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021.  BabyBERTa: Learning more grammar with small-scale child-directed language.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 624–646, Online. Association for Computational Linguistics.
  * Hupkes et al. (2023) Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023.  A taxonomy and review of generalization research in nlp.  _Nature Machine Intelligence_ , 5:1161–1174.
  * Jumelet et al. (2021) Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021.  Language models use monotonicity to assess NPI licensing.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4958–4969, Online. Association for Computational Linguistics.
  * Jumelet and Hupkes (2018) Jaap Jumelet and Dieuwke Hupkes. 2018.  Do language models understand anything? on the ability of LSTMs to understand negative polarity items.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 222–231, Brussels, Belgium. Association for Computational Linguistics.
  * Kuribayashi et al. (2020) Tatsuki Kuribayashi, Takumi Ito, Jun Suzuki, and Kentaro Inui. 2020.  Language models as an alternative evaluator of word order hypotheses: A case study in Japanese.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 488–504, Online. Association for Computational Linguistics.
  * Lakretz et al. (2021) Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021.  Mechanisms for handling nested dependencies in neural-network language models and humans.  _Cognition_ , 213:104699.  Special Issue in Honour of Jacques Mehler, Cognition’s founding editor.
  * Lapata et al. (1999) Maria Lapata, Scott McDonald, and Frank Keller. 1999.  Determinants of adjective-noun plausibility.  In _Ninth Conference of the European Chapter of the Association for Computational Linguistics_ , pages 30–36, Bergen, Norway. Association for Computational Linguistics.
  * Lesci et al. (2024) Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. 2024.  Causal estimation of memorisation profiles.
  * Leung et al. (2020) Jun Yen Leung, Guy Emerson, and Ryan Cotterell. 2020.  Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4016–4028, Online. Association for Computational Linguistics.
  * Levshina et al. (2023) Natalia Levshina, Savithry Namboodiripad, Marc Allassonnière-Tang, Mathew Kramer, Luigi Talamo, Annemarie Verkerk, Sasha Wilmoth, Gabriela Garrido Rodriguez, Timothy Michael Gupton, Evan Kidd, Zoey Liu, Chiara Naccarato, Rachel Nordlinger, Anastasia Panova, and Natalia Stoynova. 2023.  Why we need a gradient approach to word order.  _Linguistics_ , 61(4):825–883.
  * Linzen and Baroni (2021) Tal Linzen and Marco Baroni. 2021.  Syntactic structure from deep learning.  _Annual Review of Linguistics_ , 7(Volume 7, 2021):195–212.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * MacDonald (2013) Maryellen C MacDonald. 2013.  How language production shapes language form and comprehension.  _Frontiers in psychology_ , 4:226.
  * Malkin et al. (2021) Nikolay Malkin, Sameera Lanka, Pranav Goel, and Nebojsa Jojic. 2021.  Studying word order through iterative shuffling.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10351–10366, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Martin (1969) J.E. Martin. 1969.  Some competence-process relationships in noun phrases with prenominal and postnominal adjectives.  _Journal of Verbal Learning and Verbal Behavior_ , 8(4):471–480.
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.
  * Merrill et al. (2024) William Merrill, Noah A. Smith, and Yanai Elazar. 2024.  Evaluating $n$-gram novelty of language models using rusty-dawg.
  * Misra and Mahowald (2024) Kanishka Misra and Kyle Mahowald. 2024.  Language models learn rare phenomena from less rare phenomena: The case of the missing aanns.  _CoRR_ , abs/2403.19827.
  * Ney (1981) James W. Ney. 1981.  Optionality and choice in the selection of verb complements in english.  _WORD_ , 32(2):133–152.
  * Oh et al. (2024) Byung-Doh Oh, Shisen Yue, and William Schuler. 2024.  Frequency explains the inverse correlation of large language models’ size, training data amount, and surprisal’s fit to reading times.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2644–2663, St. Julian’s, Malta. Association for Computational Linguistics.
  * O’Donnell (2015) Timothy J. O’Donnell. 2015.  _Productivity and Reuse in Language: A Theory of Linguistic Computation and Storage_.  The MIT Press.
  * Patil et al. (2024) Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie Wang, Clevis Willrich, and Shane Steinert-Threlkeld. 2024.  Filtered corpus training (fict) shows that language models can generalize from indirect evidence.
  * Prashanth et al. (2024) USVSN Sai Prashanth, Alvin Deng, Kyle O’Brien, Jyothir S V au2, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A. Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, Katherine Lee, and Naomi Saphra. 2024.  Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon.
  * Scontras (2023) Gregory Scontras. 2023.  Adjective ordering across languages.  _Annual Review of Linguistics_ , 9:357–376.
  * Scontras et al. (2019) Gregory Scontras, Judith Degen, and Noah Goodman. 2019.  On the grammatical source of adjective ordering preferences.  _Semantics and Pragmatics_ , 12:1–21.
  * Scontras et al. (2017) Gregory Scontras, Judith Degen, and Noah D. Goodman. 2017.  Subjectivity Predicts Adjective Ordering Preferences.  _Open Mind_ , 1(1):53–66.
  * Scott (2002) Gary-John Scott. 2002.  _Stacked Adjectival Modification and the Structure of Nominal Phrases_.
  * Sinha et al. (2021) Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021.  Masked language modeling and the distributional hypothesis: Order word matters pre-training for little.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2888–2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sproat and Shih (1991) Richard Sproat and Chilin Shih. 1991.  _The Cross-Linguistic Distribution of Adjective Ordering Restrictions_. Springer Netherlands, Dordrecht.
  * Suijkerbuijk et al. (2023) Michelle Suijkerbuijk, Peter de Swart, and Stefan L. Frank. 2023.  The learnability of the wh-island constraint in Dutch by a long short-term memory network.  In _Proceedings of the Society for Computation in Linguistics 2023_ , pages 321–331, Amherst, MA. Association for Computational Linguistics.
  * Sweet (1898) H. Sweet. 1898.  _A New English Grammar, Logical and Historical_.  Number pt. 2 in A New English Grammar, Logical and Historical. Clarendon Press.
  * Teodorescu (2006) Alexandra Teodorescu. 2006.  Adjective ordering restrictions revisited.  In _25th West Coast Conference on Formal Linguistics_ , pages 399–407. Cascadilla Proceedings Project.
  * Trotzke and Wittenberg (2019) Andreas Trotzke and Eva Wittenberg. 2019.  Long-standing issues in adjective order and corpus evidence for a multifactorial approach.  _Linguistics_ , 57(2):273–282.
  * Truswell (2009) Robert Truswell. 2009.  Attributive adjectives and nominal templates.  _Linguistic Inquiry_ , 40(3):525–533.
  * Van Der Wal et al. (2022) Oskar Van Der Wal, Jaap Jumelet, Katrin Schulz, and Willem Zuidema. 2022.  The birth of bias: A case study on the evolution of gender bias in an English language model.  In _Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)_ , pages 75–75, Seattle, Washington. Association for Computational Linguistics.
  * Vecchi et al. (2017) Eva Maria Vecchi, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2017.  Spicy adjectives and nominal donkeys: Capturing semantic deviance using compositionality in distributional spaces.  _Cognitive science_ , 41 1:102–136.
  * Vecchi et al. (2013) Eva Maria Vecchi, Roberto Zamparelli, and Marco Baroni. 2013.  Studying the recursive behaviour of adjectival modification with compositional distributional semantics.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Vendler (1968) Z. Vendler. 1968.  _Adjectives and Nominalizations_.  Papers on formal linguistics. Mouton.
  * Warstadt et al. (2023) Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023.  Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora.  In _Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning_ , pages 1–34, Singapore. Association for Computational Linguistics.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020.  BLiMP: The benchmark of linguistic minimal pairs for English.  _Transactions of the Association for Computational Linguistics_ , 8:377–392.
  * Wilcox et al. (2018) Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018.  What do RNN language models learn about filler–gap dependencies? In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 211–221, Brussels, Belgium. Association for Computational Linguistics.
  * Wilson and Frank (2023) Michael Wilson and Robert Frank. 2023.  Inductive bias is in the eye of the beholder.  In _Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP_ , pages 152–162, Singapore. Association for Computational Linguistics.
  * Wulff (2003) Stefanie Wulff. 2003.  A multifactorial corpus analysis of adjective order in english.  _International Journal of Corpus Linguistics_ , 8:245–282.
  * Wulff and Gries (2015) Stefanie Wulff and Stefan Gries. 2015.  Prenominal adjective order preferences in chinese and german l2 english: A multifactorial corpus study.  _Linguistic Approaches to Bilingualism_ , 5.

##  Appendix A Complementary Results

###  A.1 CAP*X Examples

We provide a sample of CAP examples in Table 2, categorised by the contextual
and isolated AOP scores of Pythia-12b.

|  CAP Sentence | AOP$(\bullet)$ | AOP$(\bullet|c)$
---|---|---|---
CAP order always preferred
1. |  In solution, lead(II) hydroxide is a somewhat weak base, forming lead(II) ion, Pb, under weakly acidic conditions. | $16.5$ | $18.8$
2. |  It has the only deep water port in Cambodia. | $16.9$ | $15.4$
3. |  The greatest social reforms in Denmark are certainly the work of the last half-century. | $15.4$ | $16.3$
4. |  Icterids are unusual in songbirds because they have considerable sexual dimorphism. | $16.9$ | $14.5$
Swapped order always preferred
5. |  Couches are usually bought in a set together with cushions, which give them a bouncey and decorative comfortable touch. | $-7.1$ | $-6.8$
6. |  No, its 10 feet tall, and it’s a red big monster, demon like. | $-7.3$ | $-5.9$
7. |  It does this by following the grammatical basic rules of syntax. | $-5.8$ | $-6.0$
8. |  The daily average attendance from January through November 2010 was 22,133 people a day. | $-5.8$ | $-5.7$
Context improves AOP
9. |  Being a spherical 3-manifold, it is the only homology 3-sphere, besides the 3-sphere itself, with a finite fundamental group. | $0.9$ | $16.8$
10. |  Since the 2015 Styria municipal structural reform, it is part of the municipality Birkfeld. | $-0.5$ | $15.3$
11. |  I won it at a fair last night. | $0.2$ | $15.4$
12. |  I say that the man who does not play whist lays up a sad old age for himself. | $2.6$ | $17.5$
Context worsens AOP
13. |  This is an ability which may have been present in their last common ancestor in the Archaean. | $12.3$ | $4.0$
14. |  Jacob, you’ve had a complex partial seizure, which can cause psychosis, including religious psychosis. | $7.3$ | $0.3$
15. |  He was also a right-arm medium fast bowler with three wickets in test matches to his credit. | $6.2$ | $-0.6$
16. |  He wheeled round sharply, and distinguished her lying with helpless outspread arms on the couch. | $4.0$ | $-2.8$
Table 2: Most salient examples of the four quadrants in Figure 6.

###  A.2 Correlations to Cognitive Predictors

Correlations during training of AOP-$\Delta$ scores (Pythia-1.4b), with scores
derived from various cognitive predictors.

###  A.3 Relative Number of Bigram Occurrences

Relative sizes of the 4 splits of CAP based on the number of bigram
occurrences seen at each point in training. Note that an additional constraint
on each split is that the swapped adjective order has not been seen at all,
which explains why the 4 splits do not sum up to 1.

###  A.4 One-shot AOP

In Figure 7 we plot the AOP-$\Delta$ scores for the 27 adjective pairs that
are seen exactly once in the first 10% of training.

###  A.5 Context Improvement Ratios

The relative number of items that are affected by context with respect to
isolated AOP-$\Delta$ scores for each model size. These ratios correspond to
the four quadrants of Figure 6A.

Figure 7:  The AOP-$\Delta$ scores (Pythia-1.4b) across time for the subset of
adjective pairs that are seen exactly once up to the 14,000th batch. Each
curve is centered around the point where the adjective pair was encountered.
The dotted line denotes the average AOP-$\Delta$ across all CAP items.


On irrationality of Euler’s constant and related asymptotic formulas

Shekhar Suman
M.Sc., I.I.T. Delhi, India

Email<EMAIL_ADDRESS>
Abstract: By defining

$I_{n}:=\int_{0}^{1}\int_{0}^{1}\frac{(x(1-x)y(1-y))^{n}}{(1-xy)(-\log xy)}\
dxdy$

Sondow (see [2]) proved that

$I_{n}=\binom{2n}{n}\gamma+L_{n}-A_{n}$

We prove asymptotic formula for $L_{n}$ and $A_{n}$ as $n\to\infty$,

$L_{n}=\binom{2n}{n}\left(\log\left({\frac{{3n}}{2}}\right)+\mathcal{O}\\!\left({\frac{1}{n}}\right)\right)$

and

$A_{n}\sim\frac{4^{n}}{\sqrt{\pi n}}\left(\gamma+\ln\frac{3}{2}+\ln n\right)$

Using the sufficient condition for irrationality criteria of Euler’s constant
due to Sondow, we prove that $\gamma$ is irrational.
Keywords and Phrases: Euler’s constant, Harmonic number, asymptotic equality,
rising and falling factorial, Stirling numbers, Beta function
2020 Mathematics Subject Classification: .
1\. Introduction and Definitions The Euler’s constant is defined by the limit,

$\gamma:=\lim_{n\to\infty}(H_{n}-\log n)$ (1)

where $H_{n}=\sum_{k=1}^{n}\frac{1}{k}$ is the $n$th Harmonic number. Euler’s
constant has a double integral representation (see [1]),

$\gamma=\int_{0}^{1}\int_{0}^{1}\frac{1-x}{(1-xy)(-\log xy)}\ dxdy$ (2)

Sondow (see [2]) gave criteria for irrationality of Euler’s constant where he
defined

$I_{n}:=\int_{0}^{1}\int_{0}^{1}\frac{(x(1-x)y(1-y))^{n}}{(1-xy)(-\log xy)}\
dxdy$ (3)

If $d_{n}=\text{LCM}(1,2,...,n)$ then Sondow proved (see [2])

$I_{n}=\binom{2n}{n}\gamma+L_{n}-A_{n}$ (4)

where

$L_{n}=d^{-1}_{2n}\log S_{n}$ (5)
$S_{n}=\prod_{k=1}^{n}\prod_{i=0}^{\min{(k-1,n-k)}}\prod_{j=i+1}^{n-i}(n+k)^{\frac{2d_{2n}}{j}\binom{n}{i}^{2}}$
(6)

and

$A_{n}=\sum_{j=0}^{n}\binom{n}{j}^{2}H_{n+j}$ (7)

Clearly then we have $d_{2n}A_{n}\in\mathbb{Z}$. It was proved that a
sufficient condition for irrationality of $\gamma$ is (see [2])

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log
S_{n}\\}\neq\frac{\pi}{6\log 2}$ (8)

where $\\{x\\}$ denotes the fractional part of $x$.

2\. Main Theorems Using Laplace’s method (see [3], p.322), Sondow proved as
$n\to\infty$ (see [2]),

$I_{n}\sim\left(\frac{\pi}{6\log 2}\right)\left(\frac{1}{n4^{2n}}\right)$ (9)

$I_{n}$ can be also represented as (see [2])

$I_{n}=\sum_{v=n+1}^{\infty}\int_{v}^{\infty}\left(\frac{n!}{x(x+1)...(x+n)}\right)^{2}\
dx$ (10)

The goal of this article is to prove the following result.

Theorem : We have

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log
S_{n}\\}\neq\frac{\pi}{6\log 2}$ (11)

Proof: We prove a few Lemma:
Lemma 1: We have the following partial fraction decomposition for
$n\in\mathbb{N}$:

$\frac{1}{(x(x+1)(x+2)...(x+n))^{2}}=\sum_{k=0}^{n}\frac{a_{k}}{x+k}+\sum_{k=0}^{n}\frac{b_{k}}{(x+k)^{2}}$
(12)

where

$a_{k}=2\frac{H_{k}-H_{n-k}}{\left(k!(n-k)!\right)^{2}}$ (13)

and

$b_{k}=\frac{1}{\left(k!(n-k)!\right)^{2}}$ (14)

Proof: Since every $k$, $k=-n,...,-2,-1,0$ is a pole of order two of the given
fraction so its decomposition looks like

$\frac{1}{(x(x+1)(x+2)...(x+n))^{2}}=\sum_{k=0}^{n}\frac{a_{k}}{x+k}+\sum_{k=0}^{n}\frac{b_{k}}{(x+k)^{2}}$
(15)

Next we find $a_{k}$ and $b_{k}$: For finding $b_{k}$, we multiply each side
of (15) by $(x+k)^{2}$, simplify, and set $x=-k$ to get

$b_{k}=\left(\frac{1}{(-k)(-(k-1))\cdots(-1)(1)(2)\cdots(n-k)}\right)^{2}=\frac{1}{\left(k!(n-k)!\right)^{2}}$
(16)

Deriving a formula for $a_{k}$ is some what lengthy. The rising and falling
factorial functions may be expanded as polynomials whose coefficients are the
unsigned Stirling numbers of the first kind:

$x^{(n)}=x(x+1)\cdots(x+n-1)=\sum_{j=0}^{n}{n\brack j}x^{j}$ (17)

and

$(x)_{n}=x(x-1)\cdots(x-(n-1))=\sum_{j=0}^{n}(-1)^{n-j}{n\brack j}x^{j}$ (18)

The goal is to find coefficients of an expansion of
$\left(x^{(n+1)}\right)^{-2}$ in powers of $y\doteq x+k$. Therefore we write

$x^{(n+1)}=(y-k)(y-(k-1))\cdots(y-1)y(y+1)\cdots(y+n-k)=\frac{y^{(n-k+1)}(y)_{k+1}}{y}$
(19)

Expanding this in powers of $y$ yields

$\displaystyle x^{(n+1)}$ $\displaystyle=\frac{1}{y}\left({n-k+1\brack
0}+{n-k+1\brack 1}y+{n-k+1\brack 2}y^{2}+O(y^{3})\right)$
$\displaystyle(-1)^{k}\left(-{k+1\brack 0}+{k+1\brack 1}y-{k+1\brack
2}y^{2}+O(y^{3})\right)$ (20)

The formulas for the Stirling numbers involved are, for all $n\geq 0$,

${n+1\brack 0}=0,\quad{n+1\brack 1}=n!,\quad\text{and}\quad{n+1\brack
2}=n!H_{n},$ (21)

where $H_{n}$ is the $n^{\mathrm{th}}$ harmonic number. Therefore,

$x^{(n+1)}=(-1)^{k}k!(n-k)!\,y\left(1+(H_{n-k}-H_{k})y+O(y^{2})\right).$ (22)

So we have

$\left(x^{(n+1)}\right)^{-2}=\frac{1}{\left(k!(n-k)!\right)^{2}}\,\left(y^{-2}+2(H_{k}-H_{n-k})y^{-1}+O(1)\right).$
(23)

From this we may read off the formula for $b_{k}$ given above and this formula
for $a_{k}$:

$a_{k}=2\frac{H_{k}-H_{n-k}}{\left(k!(n-k)!\right)^{2}}.$ (24)

This completes the proof of Lemma $1$.

Lemma 2: We have the following representation for $I_{n}$,

$I_{n}=\sum_{k=1}^{\infty}\int_{0}^{\infty}(B(x+n+k,n+1))^{2}\ dx$ (25)

where $B(p,q)$ is the Beta function.
Proof: Can be derived from (10).

Lemma 3: We have the following representation for $I_{n}$,

$I_{n}=\binom{2n}{n}\gamma-\sum_{j=0}^{n}{\binom{n}{j}^{2}(2(H_{n-j}-H_{j})\log((n+j)!)+H_{n+j})}$
(26)

Proof: Since from (25)

$I_{n}=\sum_{k=1}^{\infty}\int_{0}^{\infty}(B(x+n+k,n+1))^{2}\ dx$ (27)

and we have

$(B(x+n+k,n+1))^{2}=\frac{n!^{2}}{\prod_{j=n}^{2n}(x+k+j)^{2}}$ (28)

By using partial fractions as obtained in (12) we get

$I_{n}=\sum_{k=1}^{\infty}\int_{0}^{\infty}\sum_{j=0}^{n}\binom{n}{j}^{2}\left(2\frac{H_{j}-H_{n-j}}{x+j+k+n}+\frac{1}{x+j+k+n}\right)\
dx$ (29)

Integrating and evaluating the negative integrand at $x=0$ and summing over
$k$ with a limit gives

$I_{n}=\lim_{r\to\infty}\sum_{j=0}^{n}\binom{n}{j}^{2}\left(2(H_{n-j}-H_{j})\log\left(\frac{(n+j+r)!}{(n+j)!}\right)+H_{n+j+r}-H_{n+j}\right)$
(30)

It is well known that as $r\to+\infty$

$H_{n+j+r}=\log(n+j+r)+\gamma+o(1)$ (31)

and

$\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}}=\binom{2n}{n},$ (32)

we can derive the explicit formula

$I_{n}=\binom{2n}{n}\gamma-\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}(2(H_{n-j}-H_{j})\log((n+j)!)+H_{n+j})}$
(33)

provided that

$\mathop{\lim}\limits_{r\to+\infty}\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\left({2(H_{n-j}-H_{j})\log((n+j+r)!)+\log(n+j+r)}\right)}=0.$
(34)

Denote the expression under the limit by $S_{n}(r)$. Performing the change of
summation index $j\to n-j$ and taking the average with the original
expression, we find

$S_{n}(r)=\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\left({(H_{n-j}-H_{j})\log\frac{{(n+j+r)!}}{{(2n-j+r)!}}+\frac{1}{2}\log((2n-j+r)(n+j+r))}\right)}.$
(35)

Stirling’s formula and the Maclaurin series of the logarithm then yields

$S_{n}(r)=(\log
r)\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\left({(H_{n-j}-H_{j})(2j-n)+1}\right)}+o(1)$
(36)

as $r\to+\infty$. Therefore, it remains to show that

$\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}((H_{n-j}-H_{j})(2j-n)+1)}=0$ (37)

for all $n\geq 1$. Because of the symmetry in $H_{n-j}-H_{j}$, this may be
further simplifed to the claim that

$\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}(2j(H_{n-j}-H_{j})+1)}=0$ (38)

for all $n\geq 1$. Computer algebra software confirms this for
$n=1,2,3\ldots,200$. This completes the proof for Lemma 3.

Comparing equations (4) and (26)

$L_{n}=-\sum_{j=0}^{n}{\binom{n}{j}^{2}(2(H_{n-j}-H_{j})\log(n+j)!)}$ (39)

Lemma 4: We have the following asymptotic formula for $L_{n}$ and $A_{n}$ as
$n\to\infty$,

$L_{n}=\binom{2n}{n}\left(\log\left({\frac{{3n}}{2}}\right)+\mathcal{O}\\!\left({\frac{1}{n}}\right)\right)$
(40)

and

$A_{n}\sim\frac{4^{n}}{\sqrt{\pi n}}\left(\gamma+\ln\frac{3}{2}+\ln n\right)$
(41)

Proof: This answers the question about the asymptotics of $a_{n}$ provided
that the conjectured formula for $I_{n}$ in my other answer is correct. Note
that since $H_{k}=\psi(k+1)+\gamma$, we can write

$L_{n}=I_{n}+\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\psi(n+j+1)}=-\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\psi(n+j+1)}+o(1).$
(42)

The $o$-term follows from equation (9). Now by the asymptotic result

$\psi(k+1)=\log k+\mathcal{O}(k^{-1})$ (43)

we have

$\displaystyle\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\psi(n+j+1)}$
$\displaystyle=\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\log(n+j)}+\mathcal{O}(1)\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\frac{1}{{n+j+1}}}$
$\displaystyle=\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\log(n+j)}+\mathcal{O}\\!\left({\frac{1}{n}}\right)\binom{2n}{n}.$

Since $\log n\leq\log(n+j)\leq\log n+\log 2$, it follows that as $n\to+\infty$

$L_{n}\sim\binom{2n}{n}\log n$ (44)

Also by the change of summation index from $j$ to $n-j$ and taking the average
with the original expression, we find

$\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\log(n+j)}=\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\log\sqrt{(n+j)(2n-j)}}.$
(45)

Now

$\displaystyle\log\sqrt{(n+j)(2n-j)}$
$\displaystyle=\log\left({\frac{{3n}}{2}}\right)+\log\sqrt{1-\frac{4}{9}\left({\frac{1}{2}-\frac{j}{n}}\right)^{2}}$
$\displaystyle=\log\left({\frac{{3n}}{2}}\right)+\mathcal{O}(1)\left({\frac{1}{2}-\frac{j}{n}}\right)^{2}.$

Numerics suggest that as $n\to+\infty$

$\sum\limits_{j=0}^{n}{\binom{n}{j}^{2}\left({\frac{1}{2}-\frac{j}{n}}\right)^{2}}\sim\frac{1}{{8n}}\binom{2n}{n},$
(46)

This would lead to the more precise result that as $n\to+\infty$

$L_{n}=\binom{2n}{n}\left(\log\left({\frac{{3n}}{2}}\right)+\mathcal{O}\\!\left({\frac{1}{n}}\right)\right)$
(47)

Now to find asymptotics for $L_{n}$ and $A_{n}$: We use Stirling’s formula

$I_{n}=\binom{2n}{n}\gamma-\sum_{j=0}^{n}\binom{n}{j}^{2}(2(H_{n-j}-H_{j})\ln((j+n)!)+H_{n+j})=\binom{2n}{n}\gamma+L_{n}-A_{n}$
(48)

where $A_{n}=\sum_{j=0}^{n}\binom{n}{j}^{2}H_{n+j}$ As $\binom{n}{j}^{2}$
reaches a sharp maximum near $j=\frac{n}{2}$, we can choose $n$ even for a
while, $j=\frac{n}{2}+k$ and present $A_{n}$ as

$A_{n}=\sum_{k=-n/2}^{n/2}\left(\frac{n!}{\big{(}\frac{n}{2}-k\big{)}!\big{(}\frac{n}{2}+k\big{)}!}\right)^{2}H_{\frac{3n}{2}+k}$
(49)

Using the Stirling’s formula for $p!$ ( for $p\gg 1$)

$A_{n}\sim\sum_{k=-n/2}^{n/2}\frac{\sqrt{2\pi
n}\left(\frac{n}{e}\right)^{n}H_{\frac{3n}{2}+k}}{\sqrt{2\pi(\frac{n}{2}+k)}\sqrt{2\pi(\frac{n}{2}-k)}\Big{(}\frac{\frac{n}{2}+k}{e}\Big{)}^{\frac{n}{2}+k}\Big{(}\frac{\frac{n}{2}-k}{e}\Big{)}^{\frac{n}{2}-k}}$
(50)

The terms decline sharply as soon as $k$ excides $\sqrt{n}\,$ , so we can
switch from summation to integration. Given that $H_{\frac{3n}{2}+k}$ is
slowly changing function, we are allowed just to take its value at $k=0$ and
use the asymptotics

$H_{\frac{3n}{2}}=\gamma+\ln\frac{3n}{2}+O\big{(}\frac{1}{n}\big{)}$ (51)

After manipulations we get

$A_{n}\sim\frac{2\cdot 4^{n}\big{(}\gamma+\ln\frac{3}{2}+\ln
n\big{)}}{\pi\sqrt{n}}\int_{-\infty}^{\infty}e^{-4t^{2}}dt=\frac{4^{n}}{\sqrt{\pi
n}}\left(\gamma+\ln\frac{3}{2}+\ln n\right)$ (52)

Due to the fact that

$\binom{2n}{n}\sim\frac{4^{n}}{\sqrt{\pi
n}}\to\infty\,\,\text{at}\,\,n\to\infty$ (53)

and $I_{n}\to 0$ at $n\to\infty$, we conclude that

$L_{n}\sim\frac{4^{n}}{\sqrt{\pi n}}\left(\ln\frac{3}{2}+\ln n\right)$ (54)

This proves Lemma $4$.

Now we are ready to prove the Theorem: Since $\log S_{n}>0$ (see [2]) and
$0\leq\\{\log S_{n}\\}<1$ so we get in LHS of $(8)$

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log S_{n}\\}\geq 0$
(55)

Since by (5)

$\log S_{n}=d_{2n}L_{n}$ (56)

so we have

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log
S_{n}\\}=\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{d_{2n}L_{n}\\}$
(57)

By (47), we can write (57) as

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log
S_{n}\\}=\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\left\\{d_{2n}\binom{2n}{n}\left(\log\left({\frac{{3n}}{2}}\right)+\mathcal{O}\\!\left({\frac{1}{n}}\right)\right)\right\\}$
(58)

Since by Prime number theorem, as $n\to\infty$, $d_{2n}\sim e^{2n}$ and we
have

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log
S_{n}\\}=\lim_{n\to\infty}\left(\frac{4^{2n}n}{e^{2n}}\right)\left\\{e^{2n}\binom{2n}{n}\left(\log\left({\frac{{3n}}{2}}\right)+\mathcal{O}\\!\left({\frac{1}{n}}\right)\right)\right\\}$
(59)

So we claim that

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log S_{n}\\}=0$ (60)

Mathematica also hints the answer that the above limit in (60) is $0$. So we
have

$\lim_{n\to\infty}\left(\frac{4^{2n}n}{d_{2n}}\right)\\{\log
S_{n}\\}\neq\frac{\pi}{6\log 2}$ (61)

This settles the proof of the Theorem.

Acknowledgement The authors are thankful to the Referee and the Editor for
their useful comments and processing of the article.

References

[1] Sondow, J., Double Integrals for Euler’s Constant and $\ln\frac{4}{\pi}$
and an Analog of Hadjicostas’s Formula, Amer. Math. Monthly 112 61-65, 2005.

[2] Sondow, J., Criteria for irrationality of Euler’s constant ,Proceedings of
the American Mathematical society Volume 131, Number 11, p. 3335–3344, 2003 .

[3] N. Bleistein and R. Handelsman, Asymptotic expansion of integrals, Holt,
Rinehart and Winston, 1975 .