# ConStat: Performance-Based Contamination Detection in Large Language Models

Jasper Dekoninck1, Mark Niklas Müller1,2, Martin Vechev1
Department of Computer Science1 LogicStar.ai2 ETH Zurich, Switzerland
<EMAIL_ADDRESS>

###### Abstract

Public benchmarks play an essential role in the evaluation of large language
models. However, data contamination can lead to inflated performance,
rendering them unreliable for model comparison. It is therefore crucial to
detect contamination and estimate its impact on measured performance.
Unfortunately, existing detection methods can be easily evaded and fail to
quantify contamination. To overcome these limitations, we propose a novel
definition of _contamination as artificially inflated and non-generalizing
benchmark performance_ instead of the inclusion of benchmark samples in the
training data. This perspective enables us to detect _any_ model with inflated
performance, i.e., performance that does not generalize to rephrased samples,
synthetic samples from the same distribution, or different benchmarks for the
same task. Based on this insight, we develop ConStat, a statistical method
that reliably detects and quantifies contamination by comparing performance
between a primary and reference benchmark relative to a set of reference
models. We demonstrate the effectiveness of ConStat in an extensive evaluation
of diverse model architectures, benchmarks, and contamination scenarios and
find high levels of contamination in multiple popular models including
Mistral, Llama, Yi, and the top-3 Open LLM Leaderboard models.111Code
available at https://github.com/eth-sri/ConStat.

##  1 Introduction

As large language models (LLMs) become increasingly effective at a wide range
of tasks, many companies and research institutions compete to develop better
models [2, 5, 28, 35]. To facilitate this development, a variety of benchmarks
have been proposed that allow a standardized in-depth comparison of model
performance across diverse tasks [15, 16, 26, 32].

#### Data Contamination

Modern LLMs are trained on vast amounts of internet-sourced data, raising the
risk of unintentionally including benchmark samples in the training set. Such
_data contamination_ can lead to artificially inflated benchmark performance
that does not accurately reflect a model’s true ability to generalize to
unseen tasks. However, model providers argue that the impact of this
contamination on model performance is negligible [2, 14, 35] and the enormous
size of current training sets almost guarantees contamination to some extent.
This casts doubt on the relevance of this traditional definition of
contamination in the context of LLMs.

#### This Work: A New Perspective on Data Contamination

We propose a new perspective on contamination, defining it based on its effect
on model performance rather than its cause. Specifically, we _define
contamination as artificially inflated, non-generalizing performance_ , i.e.,
we say a model is contaminated if and only if its performance relative to
other models is significantly higher on the original benchmark than on a
similar reference benchmark. This definition captures the essence of the
contamination problem, i.e., performance measurements becoming unreliable for
model comparisons. Furthermore, it enables principled detection methods that
are robust against evasion attacks by malicious providers as this would
require generalizing performance improvements.

#### Traditional Contamination Detection

Existing contamination detection methods [18, 23, 24, 30, 33, 36, 39, 40, 48]
aim to detect the inclusion of benchmark samples in the training data as a
measure of contamination. However, these approaches show limited success,
cannot quantify the contamination’s effect on model performance, and have to
make strict assumptions about the contamination process, making them easy to
evade [17].

1\. Define Problem$M_{1}$$M_{2}$Benchmark $D$72.1%72.1%Which model is
contaminated?2\. Select ReferencesReference
Models$M_{\text{ref},1}$$M_{\text{ref},2}$$M_{\text{ref},3}$Reference
BenchmarkRephrased $D_{\text{ref}}\approx D$Synthetic $D_{\text{ref}}\sim
D$Similar $D_{\text{ref}}\neq D$3\. Compute Test4\. Extract
Results$\delta_{1}=35.1\pm 3.1$$p=0.01\%$$\delta_{2}=4.1\pm 5.1$$p=30\%$

Figure 1: Overview of our method. We first select models to check for
contamination, then select reference models and benchmarks, and finally
compute ConStat to detect and quantify contamination.

#### This Work: A Statistical Test for Contamination

In contrast, we leverage our novel performance-based definition of data
contamination to propose a statistical contamination test called ConStat,
illustrated in Fig. 1. Given a target model ($M_{1}$ or $M_{2}$) to check for
contamination (first step in Fig. 1), we select a set of reference models for
performance comparison and a reference benchmark $D_{\text{ref}}$ that is
similar to the original benchmark $D$ (second step). This reference benchmark
can be a rephrased version of the original benchmark, a synthetic benchmark
generated from the same distribution, or a different benchmark measuring
performance on the same task. We then evaluate the reference models on both
benchmarks $D$ and $D_{\text{ref}}$ and fit the difficulty correction function
$H_{D_{\text{ref}}}$ describing the relation between performance on the
reference and original benchmarks (blue curve). By evaluating
$H_{D_{\text{ref}}}$ at the target model’s performance on the reference
benchmark, we predict its expected performance on the original benchmark
(third step). Finally, we compute the difference $\delta$ between this
expected performance and the model’s actual performance on the original
benchmark. Using bootstrapping, we obtain an estimate of the contamination
magnitude $\delta$ and a p-value that quantifies the likelihood of the
observed performance difference under the null hypothesis that the target
model is not contaminated (fourth step). In the illustrated case, model
$M_{1}$ achieves $60\%$ on the reference benchmark, which translates to an
expected performance of $37\%$ on the original benchmark. However, the
measured performance of $72\%$ indicates a large contamination effect
$\delta_{1}=35\%$ and thus strong contamination with a p-value of $0.01\%$. In
contrast, model $M_{2}$ shows no signs of contamination.

#### Evaluation

We evaluate ConStat on a wide range of contamination scenarios and model
architectures, demonstrating that it is significantly more effective at
detecting contamination than any prior method. We then use ConStat to study a
range of popular open and proprietary models and find high levels of
contamination in Mistral-7b-v0.1 [28], Llama-3-70b [2], Llama-2-Instruct-70b
[42], Yi-34b [52], and a range of top Open LLM Leaderboard [7] models.

#### Key Contributions

Our key contributions are:

  * •

We propose a new performance-based definition of benchmark contamination
(Section 2).

  * •

We introduce ConStat, a statistical test that detects and quantifies
contamination in language models (Section 3).

  * •

We empirically demonstrate ConStat’s effectiveness in an extensive evaluation
across various contamination scenarios (Section 4.2).

  * •

We leverage ConStat to study a range of popular models and find contamination
for Mistral, Llama, Yi, and the top-3 Open LLM Leaderboard models (Section
4.3-Section 4.5).

##  2 Defining Contamination

Before formalizing our novel definition, we first informally contrast the
traditional, information-flow-based perspective on contamination with our
novel performance-based one.

#### Information-Flow Perspective

In traditional machine learning, contamination typically refers to any
information flow between the benchmark used for performance measurement and
model training. In the context of LLMs, this is usually restricted to the
direct inclusion of test set samples (or their semantic equivalents) in the
training dataset [36, 38, 50, 55].

However, this perspective suffers from several drawbacks. First, it does not
fully capture the core issue of contamination, which is whether it renders
test set performance an unreliable predictor of real-world performance.
Second, in the era of zero-shot learning, we aim to measure performance on
"unseen" tasks, yet we train on internet-scale data that likely contains
samples of almost any task. This makes the threshold for contamination blurry.
Third, limiting the definition to test sample inclusion neglects the
possibility of model and hyperparameter selection based on benchmark
performance as a source of contamination. Finally, even with this narrow
definition, detecting contamination without access to the training data is
challenging, which makes it easy to circumvent [17].

#### Performance Perspective

To overcome these limitations, we propose to _define contamination based on
its outcome, rather than its cause_. Informally, we define contamination as
artificially inflated performance on a benchmark that does not generalize to
real-world performance on the corresponding task, regardless of how it was
achieved. This definition aligns better with the practical implications of
contamination and enables a more principled detection method that makes
evasion difficult.

To detect contamination, we compare the performance of a model $M$ on a
benchmark $D$ to its performance on a reference benchmark $D_{\text{ref}}$,
the choice of which we will discuss later. It is crucial to account for
differences in difficulty between $D$ and $D_{\text{ref}}$. Otherwise, a
slightly harder reference benchmark $D_{\text{ref}}$ would falsely indicate
inflated performance on $D$. Thus, direct performance comparison is only valid
if the distribution over sample difficulties is the same for both benchmarks,
which is a very strong assumption that is rarely true. To address this, we
compare performances relative to a set of reference models, allowing us to
determine if a model’s performance on $D$ is significantly higher than
expected, given its difficulty. In the next section, we make this definition
more formal.

###  2.1 Formal Definition of Performance-Based Contamination

#### Reference Models

To accurately compare performance between benchmarks, we use reference models
to correct for benchmark difficulty differences. For this purpose, we consider
the set of all reliable LLMs $\mathcal{M}_{\text{ref}}$ from reputable sources
to estimate the performance distribution of uncontaminated models. Although we
cannot guarantee these models are uncontaminated, we can perform leave-one-out
contamination detection to remove suspicious models from the reference set.
Furthermore, including contaminated models in $\mathcal{M}_{\text{ref}}$ will
only make our test more conservative, making it _less_ likely for
uncontaminated models to be detected as contaminated.

#### Contamination Detection

For each benchmark $D$, we define a scoring function
$S_{D}\colon\mathcal{M}\to\mathbb{R}$ that assigns a score (e.g., accuracy) to
every model from the space of all possible language models $\mathcal{M}$.
Applied to the reference models $\mathcal{M}_{\text{ref}}$, it induces a
cumulative distribution function $F_{D}$ over the uncontaminated performance
on this benchmark.

We now use the cumulative distributions $F_{D}$ and $F_{D_{\text{ref}}}$ to
predict the performance of a model $M$ on $D$ given its performance on
$D_{\text{ref}}$. Specifically, we first map the performance on the reference
data $S_{D_{\text{ref}}}(M)$ to a percentile
$q=F_{D_{\text{ref}}}(S_{D_{\text{ref}}}(M))$ and then map this percentile to
the corresponding performance on the original benchmark $F_{D}^{-1}(q)$ using
the percentile function $F_{D}^{-1}$. To simplify notation, we define the
hardness correction function $H_{D_{\text{ref}}}\colon\mathbb{R}\to\mathbb{R}$
as $H_{D_{\text{ref}}}=F_{D}^{-1}\circ F_{D_{\text{ref}}}$. This allows us to
estimate the effect of contamination on the model’s performance as
$S_{D}(M)-H_{D_{\text{ref}}}(S_{D_{\text{ref}}}(M))$ and gives our formal
definition of contamination:

######  Definition 1 ($\delta$-Contamination).

A model $M\in\mathcal{M}$ is $\delta$-contaminated on a benchmark $D$ with
respect to a reference benchmark $D_{\text{ref}}$ if
$S_{D}(M)-H_{D_{\text{ref}}}(S_{D_{\text{ref}}}(M))>\delta$.

###  2.2 Types of Contamination

Depending on the choice of reference benchmark $D_{\text{ref}}$, we can
measure different types of contamination, depending on how poorly the inflated
performance generalizes.

_Syntax-Specific Contamination_ occurs when the model fails to generalize to
semantically equivalent samples. That is, the model has memorized the exact
samples in the benchmark, and its performance drops as soon as the wording
changes. We therefore consider it to be the worst kind of contamination. To
measure syntax-specific contamination we create our reference benchmark
$D_{\text{ref}}$ by rephrasing the samples in the original benchmark $D$ to
obtain a semantically equivalent benchmark.

_Sample-Specific Contamination_ occurs when the model fails to generalize to
new samples from the benchmark distribution. That is, while the model
generalizes to samples that are semantically equivalent to those in the
original benchmark, it does not generalize to new samples from the same
distribution. To accurately measure sample-specific contamination, we would
preferably generate samples for $D_{\text{ref}}$ following the same steps used
to produce $D$. As this is often infeasible in practice, we instead generate
synthetic samples for $D_{\text{ref}}$ by querying a strong LLM using few-shot
prompting and varying the provided few-shot examples to increase diversity.

_Benchmark-Specific Contamination_ occurs when the model fails to generalize
to different benchmarks that aim to measure performance on the same task. That
is, the model generalizes to new samples from the original benchmark
distribution but does not generalize to closely related benchmarks. To measure
benchmark-specific contamination we create (or select) a different benchmark
$D_{\text{ref}}$ (e.g., MathQA) that aims to measure performance on the same
task as $D$ (e.g., GSM8k). We note that benchmark-specific contamination is by
far the least severe type of contamination. Further, while strong sensitivity
to the exact benchmark is undesirable, it is important to recognize that even
small differences between benchmarks can impact model performance. Therefore,
benchmark-specific contamination requires a more nuanced interpretation that
takes into account these differences.

##  3 ConStat: A Statistical Test for Detecting Contamination

We now present ConStat, a novel method for detecting contamination as defined
in Section 2 by computing confidence bounds on the estimated contamination
effect using a statistical test.

#### Reference Models

To approximate the underlying distribution of reference models
$\operatorname{\mathcal{M}}_{\text{ref}}$, we select a diverse sample of $m$
models
$\tilde{\mathcal{M}}_{\text{ref}}=\\{M_{\text{ref},1},...,M_{\text{ref},m}\\}\subset\operatorname{\mathcal{M}}_{\text{ref}}$.
We additionally include an inherently uncontaminated random-guessing model to
extend the coverage of our reference set.

#### Null Hypothesis

To rigorously test for contamination, we derive a null hypothesis based on our
definition of contamination. The null hypothesis is the assumption that the
model $M$ is not contaminated, meaning its actual score on the original data
is at most $\delta$ worse than the predicted one:
$S_{D}(M)-H_{D_{\text{ref}}}(S_{D_{\text{ref}}}(M))\leqslant\delta$ where
$\delta\in\mathbb{R}_{\geq 0}$ can be chosen freely.

#### Estimating the Hardness Correction Function

To compute the hardness correction function $H_{D_{\text{ref}}}$, we first
estimate the CDFs $F_{D}$ and $F_{D_{\text{ref}}}$ as the empirical CDFs
$\tilde{F}_{D}$ and $\tilde{F}_{D_{\text{ref}}}$, respectively. To this end,
let $i_{1},...,i_{n}$ be an index such that
$S_{D}(M_{\text{ref},i_{k}})\leqslant S_{D}(M_{\text{ref},i_{k+1}})$. We
obtain the CDF $\tilde{F}_{D}$ as

$\tilde{F}_{D}(x)=\begin{cases}0&\text{if}\;\;x<S_{D}(M_{i_{1}})\\\
k/n&\text{if}\;\;S_{D}(M_{i_{k}})\leqslant x<S_{D}(M_{i_{k+1}})\\\
1&\text{if}\;\;S_{D}(M_{i_{n}})\leqslant x\end{cases}.$ (1)

Similarly, $\tilde{F}_{D_{\text{ref}}}$ can be obtained from an index
$j_{1},...,j_{n}$ such that $S_{D_{\text{ref}}}(M_{\text{ref},j_{k}})\leqslant
S_{D_{\text{ref}}}(M_{\text{ref},j_{k+1}})$. Using Eq. 1, we find that
$H_{D_{\text{ref}}}(S_{D_{\text{ref}}}(M_{j_{k}}))=S_{D}(M_{i_{k}})$. Applying
the empirical CDFs directly to other points $x\in[0,1]$ would result in a step
function estimate of $H_{D_{\text{ref}}}$, leading to an overly rough
approximation of the hardness correction function. Thus, we compute the
approximate hardness function $\tilde{H}_{D_{\text{ref}}}$ by fitting the
points $(S_{D_{\text{ref}}}(M_{j_{k}}),S_{D}(M_{i_{k}}))$ using a smoothing
spline, minimizing the following loss function:

$\sum_{k=1}^{n}\big{(}S_{D}(M_{i_{k}})-\hat{H}_{D_{\text{ref}}}(S_{D_{\text{ref}}}(M_{j_{k}}))\big{)}^{2}+\lambda\int_{0}^{1}\hat{H}_{D_{\text{ref}}}^{\prime\prime}(x)^{2}\,dx$
(2)

where $\lambda$ is a smoothing parameter that is chosen using generalized
cross-validation [45].

#### Significance Estimation

We determine the statistical significance for rejecting the null hypothesis
via bootstrapping over both the reference models and the samples in the
benchmark, using pivotal intervals [41] to correct for uncertainty in the
bootstrapping process. By bootstrapping the models, we consider the effect of
our reference model selection
$\tilde{\operatorname{\mathcal{M}}}_{\text{ref}}$. By bootstrapping the
samples, we additionally include the error in our estimation of the scores
themselves. Thus, given the estimate
$\hat{\delta}=S_{D}(M)-\hat{H}_{D_{\text{ref}}}(S_{D_{\text{ref}}}(M))$ and
corresponding bootstrap estimates $\hat{\delta}_{1},...,\hat{\delta}_{n}$, we
compute the p-confidence lower bound for $\delta$ as
$\hat{\delta}_{1-p}=2\hat{\delta}-\hat{\delta}^{\prime}_{1-p}$ where
$\hat{\delta}^{\prime}_{q}$ is the $q$-quantile of
$\hat{\delta}_{1},...,\hat{\delta}_{n}$. From this, we obtain the p-value by
inverting this lower bound with respect to $q$. Thus, we reject the null
hypothesis for a given $\delta$ with significance level $p$ by computing the
lowest $p$ such that
$2\hat{\delta}-\hat{\delta}^{\prime}_{1-p}\geqslant\delta$.

#### Threat Model

In accordance with [17], we briefly outline the threat model assumed by
ConStat. Since we only require the ability to measure the performance of the
model on the benchmark, our method is a black-box benchmark-level detection
method that is robust to semantic preserving operations. Furthermore, we make
no additional assumptions on potential metadata contamination. However, we do
rely on the existence of reference models which we can use to estimate the
performance of uncontaminated models. Notably, however, we do not assume these
reference models to have a similar performance or architecture as the model we
wish to test.

##  4 Evaluation

In this section, we evaluate ConStat empirically. We first demonstrate
ConStat’s effectiveness, showing it outperforms prior methods in detecting and
quantifying contamination across a range of intentionally contaminated models
(Section 4.2). Next, we investigate the contamination of our chosen reference
models (Section 4.3), popular model families (Section 4.4), and top Open LLM
Leaderboard models (Section 4.5). Further, we conduct an ablation study in a
simulated environment in App. B to validate several design choices of ConStat.

###  4.1 Experimental Setup

#### Reference Models

We select 20 models from reputable providers, including Meta’s Llama model
families [2, 42], Microsoft’s Phi-2 [27] and Phi-3 [1], Google’s Gemma-1.1
[21], several Mistral models [28], Falcon-7b [3], and the fully open-source
OLMo [25]. A detailed overview of these reference models is available in App.
C.

#### Benchmarks

We select a diverse set of four of the most popular LLM benchmarks to evaluate
ConStat: GSM8k [16] is a benchmark for mathematical reasoning, ARC-Challenge
[15] is a multiple-choice benchmark for science questions, MMLU [26] is a
multiple-choice general purpose benchmark and Hellaswag [53] is a dataset for
commonsense natural language inference. Due to computational constraints, we
limit the number of samples in each benchmark to $2000$.

#### Reference Benchmarks

To generate reference data for syntax-specific and sample-specific
contamination we query GPT-4-Turbo [35] to rephrase samples from the original
benchmark and generate new synthetic samples. We generate around 1000
synthetic samples per benchmark and refer to App. C for further details on the
generation process. To detect benchmark-specific contamination, we select
appropriate reference benchmarks that measure performance on the same task:
for GSM8k, we use MathQA [4], for ARC-Challenge we use SCIQ [46], and for
Hellaswag we use the Lambada-OpenAI benchmark [37]. For MMLU, we did not
select any reference benchmark and thus measured only syntax- and sample-
specific contamination.

#### Evaluation

For evaluation, we use the LM Evaluation Harness [20] in a $5$-shot setting.
We report estimated effects $\hat{\delta}$ along with the p-value for the null
hypothesis that the effect $\delta$ is less than $0$.

###  4.2 Validating Contamination Detection with ConStat in a Controlled
Setting

In this section, we demonstrate the effectiveness of ConStat in detecting and
quantifying contamination in a controlled setting and compare it to multiple
baselines. For this purpose, we finetune both Llama-2-Instruct-7b and Phi-2
using a variety of hyperparameters and contamination scenarios on each
benchmark separately. We vary the number of epochs, the learning rate, the
portion of contaminated training samples, whether or not few-shot examples are
used during fine-tuning, and whether the model is trained on the original
benchmark samples or on rephrased data. For more details, we refer to App. C.
We trained a total of $70$ models, $9$ of which finished training at a loss
spike and were therefore excluded from further analysis. $46$ of the remaining
models were trained on the actual benchmark and should therefore exhibit both
syntax- and sample-specific contamination. The rest were trained on rephrased
benchmark data and should therefore only exhibit sample-specific
contamination. To quantify the true sample-specific contamination effect, we
only use half of each benchmark for contamination and measure the performance
gap to the other half.

#### Detecting Contamination

We first check whether ConStat can accurately detect the presence of
contamination. We compare ConStat against several baselines [12, 34, 39, 40,
51] that aim to detect contamination based on the presence of benchmark
samples in the training data. Most of these baselines [12, 34, 40, 51] require
a detection threshold to be chosen for each model and benchmark separately.
This tuning process requires uncontaminated samples, making it impossible to
apply these methods in practice. For comparison to ConStat, we tuned these
thresholds on the uncontaminated half of the benchmark, which is the most
ideal (but unrealistic) scenario. We extract a p-value for these baselines by
bootstrapping the samples in the benchmarks and checking how often
TPR@$1\%$FPR is bigger than $1\%$. Models are considered contaminated for any
method if $p<0.05$. The only baseline applicable in a realistic setting is Shi
[39] and we use their recommendation to consider a model contaminated if the
score returned by their method is above $0.85$.

Table 1: Percentage of syntax- and sample-specific contaminated models
detected by several methods.

Method | Syntax [%] | Sample [%]
---|---|---
Carlini et al. [12] | $76.1^{*}$ | $65.6^{*}$
Mireshghallah et al. [34] | $76.1^{*}$ | $68.9^{*}$
Yeom et al. [51] | $78.3^{*}$ | $67.2^{*}$
Shi et al. [40] | $84.8^{*}$ | $70.5^{*}$
Shi [39] | $21.7$ | $16.4$
ConStat | $\bf{89.1}$ | $\bf{98.4}$

* indicates that the method needs unrealistic access to uncontaminated samples for hyperparameter selection.

Results in Table 1 show that ConStat significantly outperforms all other
methods without needing prior knowledge of uncontaminated samples. In
particular, we find that ConStat can detect $89\%$ of syntax-specifically
contaminated models, while the best baseline achieves only $85\%$. The gap
widens further for sample-specific contamination, where ConStat detects $98\%$
of contaminated models, while the best baseline only detects $71\%$. The only
baseline that can be applied in a realistic setting, Shi [39], performs
significantly worse than ConStat.

We thus conclude that ConStat is the only contamination detection method that
can reliably detect contamination and significantly outperforms all baselines
even if they are tuned optimally using oracle access to the uncontaminated
samples.

Figure 2: Estimated $\hat{\delta}$ as a function of the true $\delta$ for the
finetuned models. $2$-sigma intervals are shown.

#### Quantifying Contamination

To evaluate ConStat’s ability to estimate the sample-specific contamination
effect, we compare its estimate to ground truth measurements on uncontaminated
samples. As shown in Fig. 2, we observe excellent predictiveness at a
coefficient of determination of $r^{2}=0.94$. The only three models that show
a significantly higher estimate than the true effect achieve a perfect score
on the contaminated samples, capping the true effect and explaining the
overestimation.

#### Detailed Analysis on GSM8k

We conduct an in-depth analysis of contaminated models finetuned on GSM8k,
referring to Section A.2 for a detailed table with all p-values. We finetuned
18 models on this benchmark, one of which remained undetected under sample-
specific contamination detection. For the detected syntax-specifically
contaminated models, we observe an average increase in $\hat{\delta}$ with a
factor of $2.28$ when transitioning from syntax-specific to sample-specific
contamination. This indicates that the models still generalize somewhat to
semantically equivalent samples. Furthermore, the models that were not
detected by the syntax-specific contamination detection are exactly those
models that were trained on rephrased data or were trained for just one epoch.
This indicates that these models can still generalize to semantically
equivalent samples. Since these scenarios are also more likely to occur in
practice, this shows that it is crucial to also consider sample-specific
contamination when applying ConStat. Finally, the model that remained
undetected by the sample-specific contamination detection was a Phi-2 model
trained with a lower learning rate. For this model, the actual contamination
effect is approximately $5\%$, which is relatively small and thus indicates
that ConStat is not missing any major contamination.

###  4.3 Contamination of Reputable Reference Models

Table 2: Contamination results for the reference models on syntax-specific, sample-specific, and benchmark-specific contamination. We only report tests for which the multiple testing corrected p-value is lower than $5\%$ and include the non-corrected p-value, the estimated effect $\hat{\delta}$, the $95\%$ lower bound of the effect $\hat{\delta}_{0.95}$ and the model performance on the benchmark. S stands for sample-specific and B for benchmark-specific contamination. All numbers are reported in percentages. Model | Benchmark | Type | Perf. [%] |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Llama-3-70b | ARC | S | $69.027\,304$ | $0.030\,000$ | $6.609\,663$ | $3.212\,481$
Mistral-7b-v0.1 | GSM8k | S | $39.044\,731$ | $0.150\,000$ | $8.245\,073$ | $4.479\,598$
Mistral-7b-v0.1 | Hellaswag | S | $83.650\,000$ | $0.240\,000$ | $3.135\,526$ | $1.271\,238$
Llama-2-Instruct-70b | Hellaswag | S | $85.550\,000$ | $0.410\,000$ | $3.368\,447$ | $1.287\,811$
Mistral-Instruct-7b-v0.2 | ARC | B | $62.457\,338$ | $0.040\,000$ | $10.619\,953$ | $5.953\,197$
Mistral-Instruct-7b-v0.2 | Hellaswag | B | $84.550\,000$ | $0.180\,000$ | $3.524\,014$ | $1.562\,859$
Phi-2 | GSM8k | B | $58.908\,264$ | $<10^{-2}$ | $36.418\,186$ | $26.464\,750$
Phi-3-Mini | GSM8k | B | $76.648\,976$ | $0.290\,000$ | $16.298\,102$ | $6.328\,980$
OLMo-Instruct-7b | GSM8k | B | $11.751\,327$ | $<10^{-2}$ | $8.863\,050$ | $4.989\,178$

To determine if our set of reference models exhibit signs of contamination, we
perform a leave-one-out analysis, where we evaluate the contamination of model
$M$ using $\tilde{\mathcal{M}}_{\text{ref}}\setminus\\{M\\}$ as reference
models. To control for performing multiple p-value tests and reduce the chance
of false positives, we apply the Benjamini-Hochberg [9] procedure per
benchmark and contamination type to control the false discovery rate at $5\%$.
We report all significant results in Table 2 and we discuss them for each type
of contamination below.

#### Syntax-Specific Contamination

As expected, we do not find syntax-specific contamination in any reference
model, i.e., none of the models fail to generalize to semantically equivalent
samples.

#### Sample-Specific Contamination

We find four instances of sample-specific contamination, all with very
significant p-values of less than $p=0.5\%$ and considerable estimated
contamination effects between $3\%$ and $8\%$. Specifically, we find
contamination of Llama-3-70b on ARC, of Mistral-7b-v0.1 and
Llama-2-Instruct-70b on Hellaswag, and Mistral-7b-v0.1 on GSM8k. We note that
the contamination of Llama-2-Instruct-70b on Hellaswag is noted by its model
provider [42], but the other model providers do not provide any contamination
report for their models.

We investigate these models further on the other benchmarks where the
corrected p-value using the Benjamini-Hochberg procedure was not significant.
We discuss these results below and refer to App. A for a full overview of
their sample-specific contamination. We find that Mistral-7b-v0.1 achieves
relatively low p-values on both remaining benchmarks ($8\%$ for ARC, $15\%$
for MMLU). Furthermore, we additionally evaluated Mistral-7b-v0.2 after
obtaining these results and found similar results for this model (see Table 17
in App. E). Therefore, we exclude Mistral-7b-v0.1 from our set of reference
models. While in particular Llama-3-70b also exhibits low p-values for other
benchmarks, none fall below $p\leqslant 1\%$. It is thus highly likely that
also Llama-3-70b and Llama-2-Instruct-70b are contaminated across several
benchmarks, but we keep both as reference models to ensure that we do not
obtain a higher false positive rate in our further analysis.

#### Benchmark-Specific Contamination

While we find several instances of benchmark-specific contamination in the
reference models, several at very low p-values ($p<0.01\%$), this requires a
more nuanced interpretation. For example, both Phi models exhibit very large
effect sizes ($>15\%$) and small p-values ($p<0.01\%$) for contamination on
GSM8k. We suspect that this is due to their reasoning-focused training process
and small model size. While GSM8k allows free text answers, giving the model
tokens to reason, MathQA is a multiple-choice benchmark that requires the
model to answer immediately and therefore gives no room for this reasoning
ability to shine.

###  4.4 Contamination of Popular Model Families

We now use ConStat to detect contamination in four popular model families,
discussing results for Qwen-1.5 [6] and Yi [52] below, while deferring
discussions of StableLM-2 [8] and InternLM-2 [11] to Section A.1.

####  Qwen-1.5

We evaluate all chat models from the Qwen-1.5 model family, with sizes 1.8b,
4b, 7b, 14b, 72b, and 110b. The only case of sample-specific contamination is
for the 4b model on GSM8k with $p<10^{-4}$ and an estimated effect of $5.4\%$.
The larger models show significant benchmark-specific contamination on ARC and
Hellaswag, with p-values smaller than $1\%$ and estimated effects between
$8\%$ and $14\%$.

####  Yi

We evaluate both the 6b and 34b parameter base models of the Yi model-family.
Only Yi-34b shows significant contamination, with sample-specific
contamination at $p<0.2\%$ and estimated effects of around $6\%$ on both ARC
and Hellaswag. We find additional sample-specific contamination on GSM8k of
around $4\%$ at a p-value of $p=6\%$ and _syntax-specific_ contamination on
Hellaswag at a p-value of $p=5\%$. Thus, we conclude that this model shows
significant contamination across multiple benchmarks.

###  4.5 Contamination of Top Open LLM Leaderboard Models

We use ConStat to investigate contamination in the top three 7B models on the
open LLM Leaderboard222Rank on Open LLM Leaderboard as of the 4th of May
2024., BarraHome/Mistroll-7b-v2.2, yam-peleg/Experiment26-7b, and
MTSAIR/multi_verse_model and find that all three models exhibit significant
benchmark-specific contamination. Specifically, all models show strong
contamination with estimated effects of $\hat{\delta}>10$% for the benchmarks
where the reference benchmark is not included in the Open LLM Leaderboard
(GSM8k, Hellaswag, and ARC). Further, all models show significant sample-
specific contamination on GSM8k with $\hat{\delta}\approx 9$%. For more
detailed results, we refer to App. A.

This inflated performance could be caused by a model selection bias, as the
Open LLM Leaderboard features thousands of models. This issue is exacerbated
by the recent trend of merging models [22, 49] where hyperparameters are
frequently selected based on their benchmark performance. We therefore urge
the community to be more cautious when selecting models from the leaderboard.

##  5 Related Work

#### Contamination Detection

Contamination detection methods can be broadly divided into two main
categories. The first category [10, 14, 19, 35, 42, 44, 50] focuses on
analyzing the training data directly to identify overlaps with the benchmarks
used for model evaluation. The second category [18, 23, 24, 30, 33, 36, 39,
40, 48] relies solely on access to the model and its predictions, aiming to
detect contamination through model queries. As noted by Dekoninck et al. [17],
some of these methods require metadata (e.g., benchmark name, canonical
ordering) to be leaked along with the benchmark samples in the training data
[23, 24, 36]. Methods that do not require metadata depend on perplexity-based
metrics to measure the model’s uncertainty on benchmark samples, but these can
be easily circumvented by training on rephrased samples [17]. It is important
to note that none of these methods can estimate the influence of contamination
and that they are outperformed by ConStat in terms of detection accuracy (see
Section 4.2).

An alternative approach is presented by Zhu et al. [56], who measure model
performance on rephrased benchmarks instead of the original benchmarks to
obtain more accurate estimates of model performance. However, their results
vary significantly across benchmarks, they do not provide a statistical
framework for contamination detection, and they only demonstrate that
evaluating on rephrased samples _partially_ recovers the results of
uncontaminated base models. Furthermore, they do not go beyond measuring
performance on rephrased benchmarks and can therefore also be evaded by
training on rephrased samples [17].

#### Reference Benchmarks

Recent studies have introduced new benchmarks designed to evaluate performance
on tasks similar to those in prior popular benchmarks and thus can be used to
estimate the degree of contamination. GSM1k [54] was developed to closely
replicate the efforts behind GSM8k and to compare model performances between
these benchmarks. However, GSM1k lacks a statistical test, and the slight
variations between GSM8k and GSM1k might partially explain the contamination
levels observed in their analysis. Another recent benchmark, SWE-bench [29],
focuses on evaluating performance on coding tasks. By comparing their results
with those of Human-Eval [13], one can visually interpret potential
contamination in Human-Eval. However, the absence of a statistical test
hinders precise contamination detection. In both scenarios, ConStat can
improve their findings, enabling accurate estimations of contamination in
existing models.

##  6 Discussion

#### Limitations

Our method estimates the effect of contamination on performance relative to a
set of reference models. Therefore, if these reference models are also
contaminated, our method only measures the effect relative to this base level
of contamination. However, our leave-one-out experiment, presented in Section
4.3, helps identify and exclude contaminated models, partially mitigating this
limitation. Furthermore, it is important to note that accurate relative
performance measurements are sufficient for both model selection and to assess
methodological improvements, which are the most important use cases of
benchmarks.

Further, our work uses an LLM to generate synthetic samples, introducing
potential distributional biases into the synthetic benchmark $D_{\text{ref}}$.
We briefly discuss these biases here. Firstly, synthetic benchmark may contain
more mislabeled samples. However, since these samples equally affect all
models, ConStat accounts for this in its difficulty correction. Secondly,
synthetic samples generated by a model are likely easier for that model itself
to solve. Therefore, contamination results for the model used to generate the
samples would be unreliable for sample-specific contamination detection.
However, these limitations are not inherent flaws of ConStat, and can be
mitigated by using more sophisticated synthetic benchmark generation
techniques.

#### Impact

Model evaluation is a crucial part of LLM development, with benchmarks playing
a key role in evaluating model performance on tasks like code generation,
question answering, and summarization. Contamination of these benchmarks can
inflate performance estimates, potentially misleading researchers and
practitioners. To address this, ConStat provides a statistical framework to
estimate the impact of contamination on model performance. This enables more
accurate evaluations and allows for the removal of suspicious models from
leaderboards, ensuring a fairer evaluation of model capabilities. Furthermore,
it is important to note that ConStat can be applied to any model, not just
LLMs, as long as the model’s performance can be measured on a benchmark.

##  7 Conclusion

We present ConStat, a statistical framework designed to detect contamination
and estimate its effect on model performance. Unlike existing methods, ConStat
is based on a novel, performance-based definition of contamination and
compares performance with various reference benchmarks to obtain a detailed
contamination analysis that distinguishes between syntax-, sample-, and
benchmark-specific contamination. We investigate ConStat’s effectiveness in an
extensive control study and demonstrate that it not only outperforms existing
methods but also, in contrast to them, does not require prior knowledge about
uncontaminated samples. Finally, we use ConStat to investigate contamination
in popular models and find, among others, very high levels of contamination in
Mistral-7b-v0.1 and Yi-34b and high levels of contamination in Llama-3-70b and
Llama-2-Instruct-70b.

## Acknowledgements

This work has been done as part of the EU grant ELSA (European Lighthouse on
Secure and Safe AI, grant agreement no. 101070617) and was funded in part by
the Swiss National Science Foundation (SNSF) [200021_207967]. Views and
opinions expressed are however those of the authors only and do not
necessarily reflect those of the European Union or European Commission.
Neither the European Union nor the European Commission can be held responsible
for them.

The work has received funding from the Swiss State Secretariat for Education,
Research and Innovation (SERI).

## References

  * Abdin et al. [2024] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou.  Phi-3 technical report: A highly capable language model locally on your phone, 2024.
  * AI@Meta [2024] AI@Meta.  Llama 3 model card.  2024\.  URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
  * Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The falcon series of language models: Towards open frontier models.  2023\.
  * Amini et al. [2019] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2357–2367. Association for Computational Linguistics, 2019\.  doi: 10.18653/V1/N19-1245.  URL https://doi.org/10.18653/v1/n19-1245.
  * Anil et al. [2023] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al.  Gemini: A family of highly capable multimodal models.  _CoRR_ , abs/2312.11805, 2023.  doi: 10.48550/ARXIV.2312.11805.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.  Qwen technical report.  _CoRR_ , abs/2309.16609, 2023.  doi: 10.48550/ARXIV.2309.16609.  URL https://doi.org/10.48550/arXiv.2309.16609.
  * Beeching et al. [2023] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * Bellagente et al. [2024] Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, Meng Lee, Emad Mostaque, Michael Pieler, Nikhil Pinnaparju, Paulo Rocha, Harry Saini, Hannah Teufel, Niccoló Zanichelli, and Carlos Riquelme.  Stable LM 2 1.6b technical report.  _CoRR_ , abs/2402.17834, 2024.  doi: 10.48550/ARXIV.2402.17834.  URL https://doi.org/10.48550/arXiv.2402.17834.
  * Benjamini and Hochberg [1995] Yoav Benjamini and Yosef Hochberg.  Controlling the false discovery rate: A practical and powerful approach to multiple testing.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 57(1):289–300, 1995.  doi: https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.  URL https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.2517-6161.1995.tb02031.x.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _Proc. of NeurIPS_ , 2020.
  * Cai et al. [2024] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, and et al.  Internlm2 technical report.  _CoRR_ , abs/2403.17297, 2024.  doi: 10.48550/ARXIV.2403.17297.  URL https://doi.org/10.48550/arXiv.2403.17297.
  * Carlini et al. [2021] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel.  Extracting training data from large language models.  In _30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021_ , 2021.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _CoRR_ , abs/2107.03374, 2021.  URL https://arxiv.org/abs/2107.03374.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways.  _CoRR_ , abs/2204.02311, 2022.  doi: 10.48550/arXiv.2204.02311.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the AI2 reasoning challenge.  _ArXiv preprint_ , abs/1803.05457, 2018.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _ArXiv preprint_ , abs/2110.14168, 2021.
  * Dekoninck et al. [2024] Jasper Dekoninck, Mark Niklas Müller, Maximilian Baader, Marc Fischer, and Martin T. Vechev.  Evading data contamination detection for language models is (too) easy.  _CoRR_ , abs/2402.02823, 2024.  doi: 10.48550/ARXIV.2402.02823.  URL https://doi.org/10.48550/arXiv.2402.02823.
  * Deng et al. [2023] Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan.  Investigating data contamination in modern benchmarks for large language models.  _CoRR_ , abs/2311.09783, 2023.  doi: 10.48550/ARXIV.2311.09783.
  * Dodge et al. [2021] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In _Proc. of EMNLP_ , 2021.  doi: 10.18653/v1/2021.emnlp-main.98.
  * Gao et al. [2023] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, 2023.
  * Gemma Team et al. [2024] Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al.  Gemma.  2024\.  doi: 10.34740/KAGGLE/M/3301.  URL https://www.kaggle.com/m/3301.
  * Goddard et al. [2024] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz.  Arcee’s mergekit: A toolkit for merging large language models.  _CoRR_ , abs/2403.13257, 2024.  doi: 10.48550/ARXIV.2403.13257.  URL https://doi.org/10.48550/arXiv.2403.13257.
  * Golchin and Surdeanu [2023a] Shahriar Golchin and Mihai Surdeanu.  Data contamination quiz: A tool to detect and estimate contamination in large language models.  _CoRR_ , abs/2311.06233, 2023a.  doi: 10.48550/ARXIV.2311.06233.
  * Golchin and Surdeanu [2023b] Shahriar Golchin and Mihai Surdeanu.  Time travel in llms: Tracing data contamination in large language models.  _CoRR_ , abs/2308.08493, 2023b.  doi: 10.48550/ARXIV.2308.08493.
  * Groeneveld et al. [2024] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi.  Olmo: Accelerating the science of language models.  _CoRR_ , abs/2402.00838, 2024.  doi: 10.48550/ARXIV.2402.00838.  URL https://doi.org/10.48550/arXiv.2402.00838.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _Proc. of ICLR_ , 2021.
  * Javaheripi et al. [2023] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, et al.  Phi-2: The surprising power of small language models.  https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/, 2023\.
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b.  _CoRR_ , abs/2310.06825, 2023.  doi: 10.48550/ARXIV.2310.06825.
  * Jimenez et al. [2023] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.  Swe-bench: Can language models resolve real-world github issues?  _CoRR_ , abs/2310.06770, 2023.  doi: 10.48550/ARXIV.2310.06770.  URL https://doi.org/10.48550/arXiv.2310.06770.
  * Li [2023] Yucheng Li.  An open source data contamination report for large language models.  _CoRR_ , abs/2310.17589, 2023.  doi: 10.48550/ARXIV.2310.17589.
  * Lian et al. [2023] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium".  Openorca: An open dataset of gpt augmented flan reasoning traces.  https://https://huggingface.co/Open-Orca/OpenOrca, 2023.
  * Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans.  TruthfulQA: Measuring how models mimic human falsehoods.  In _Proc. of ACL_ , 2022.  doi: 10.18653/v1/2022.acl-long.229.
  * Mattern et al. [2023] Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick.  Membership inference attacks against language models via neighbourhood comparison.  In _Findings of ACL_ , 2023.  doi: 10.18653/V1/2023.FINDINGS-ACL.719.
  * Mireshghallah et al. [2022] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri.  Quantifying privacy risks of masked language models using membership inference attacks.  In _Proc. of EMNLP_ , 2022.  doi: 10.18653/V1/2022.EMNLP-MAIN.570.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.  doi: 10.48550/arXiv.2303.08774.
  * Oren et al. [2023] Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto.  Proving test set contamination in black box language models.  _CoRR_ , abs/2310.17623, 2023.  doi: 10.48550/ARXIV.2310.17623.
  * Paperno et al. [2016] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics, 2016\.  doi: 10.18653/V1/P16-1144.  URL https://doi.org/10.18653/v1/p16-1144.
  * Sainz et al. [2023] Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre.  NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark.  In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_ , 2023.
  * Shi [2023] Weijia Shi.  Detect-pretrain-code-contamination.  https://github.com/swj0419/detect-pretrain-code-contamination, 2023\.
  * Shi et al. [2023] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer.  Detecting pretraining data from large language models.  _CoRR_ , abs/2310.16789, 2023.  doi: 10.48550/ARXIV.2310.16789.
  * Tibshirani [1984] Robert J Tibshirani.  _Bootstrap confidence intervals_.  Stanford University. Department of Statistics. Laboratory for Computational …, 1984.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288, 2023.  doi: 10.48550/arXiv.2307.09288.
  * Virtanen et al. [2020] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ , 17:261–272, 2020.  doi: 10.1038/s41592-019-0686-2.
  * Vu et al. [2023] Thuy-Trang Vu, Xuanli He, Gholamreza Haffari, and Ehsan Shareghi.  Koala: An index for quantifying overlaps with pre-training corpora.  In _Proc. of EMNLP_ , 2023.
  * Wahba [1990] Grace Wahba.  _Estimating the Smoothing Parameter_ , pages 45–65.  1990\.  doi: 10.1137/1.9781611970128.ch4.  URL https://epubs.siam.org/doi/abs/10.1137/1.9781611970128.ch4.
  * Welbl et al. [2017] Johannes Welbl, Nelson F. Liu, and Matt Gardner.  Crowdsourcing multiple choice science questions.  In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, _Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017_ , pages 94–106. Association for Computational Linguistics, 2017.  doi: 10.18653/V1/W17-4413.  URL https://doi.org/10.18653/v1/w17-4413.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In _Proc. of EMNLP_ , 2020.  doi: 10.18653/v1/2020.emnlp-demos.6.
  * Xu et al. [2024] Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu.  Benchmarking benchmark leakage in large language models, 2024.
  * Yadav et al. [2023] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal.  Ties-merging: Resolving interference when merging models.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , 2023\.  URL http://papers.nips.cc/paper_files/paper/2023/hash/1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html.
  * Yang et al. [2023] Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica.  Rethinking benchmark and contamination for language models with rephrased samples.  _CoRR_ , abs/2311.04850, 2023.  doi: 10.48550/ARXIV.2311.04850.
  * Yeom et al. [2018] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In _31st IEEE Computer Security Foundations Symposium, CSF 2018, Oxford, United Kingdom, July 9-12, 2018_ , 2018.  doi: 10.1109/CSF.2018.00027.
  * Young et al. [2024] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.  Yi: Open foundation models by 01.ai.  _CoRR_ , abs/2403.04652, 2024.  doi: 10.48550/ARXIV.2403.04652.  URL https://doi.org/10.48550/arXiv.2403.04652.
  * Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 4791–4800. Association for Computational Linguistics, 2019.  doi: 10.18653/V1/P19-1472.  URL https://doi.org/10.18653/v1/p19-1472.
  * Zhang et al. [2024] Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue.  A careful examination of large language model performance on grade school arithmetic, 2024.
  * Zhou et al. [2023] Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han.  Don’t make your LLM an evaluation benchmark cheater.  _CoRR_ , abs/2311.01964, 2023.  doi: 10.48550/ARXIV.2311.01964.
  * Zhu et al. [2023] Wenhong Zhu, Hongkun Hao, Zhiwei He, Yunze Song, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu.  CLEAN-EVAL: clean evaluation on contaminated large language models.  _CoRR_ , abs/2311.09154, 2023.  doi: 10.48550/ARXIV.2311.09154.

##  Appendix A Additional Results

We present the complete results for the experiments discussed in Section 4
here and include a discussion on the StableLM-2 and InternLM-2 model families.
We provide a table with the results for all evaluated model families where
$p<1\%$ in Table 4.

###  A.1 Discussion on InternLM-2 and StableLM-2

####  InternLM-2

We evaluated four models in the InternLM-2 model family: the models with size
1.8b and 7b, and the math-base and math models, also of 7b parameters.
Overall, we found very little evidence of contamination in these models, with
no model showing significant ($p<1\%$) sample-specific contamination. However,
we did find some evidence for benchmark-specific contamination for GSM8k and
Hellaswag for several models in the model family. Specifically, InternLM-2-7b
and InternLM-2-Math-7b show significant benchmark-specific contamination on
GSM8k with $p<0.5\%$ and estimated effects of $20\%$ and $40\%$ respectively.
The size of this effect is likely due to the same reasons as the measured
contamination in the Phi models, where the models are too small to solve
mathematical questions in one go and have been trained/finetuned to perform
chain-of-thought mathematics. The benchmark-specific contamination on
Hellaswag is present for all three 7b models in the family, with $0.5\%<p<1\%$
and estimated effects of $6\%$ to $11\%$.

####  StableLM-2

For StableLM-2, we evaluated 6 models in the model family (12b, Instruct-12b,
1.6b, Instruct-1.6b, Zephyr-3b and Alpha-7b-v2). We found only one instance of
sample-specific contamination, with the 12b model showing slight ($p=0.7\%$)
contamination for ARC. Notably, we found that all of the models in this family
show benchmark-specific contamination for GSM8k with estimated effects between
$15\%$ and $40\%$.

Table 3: Full overview of sample-specific contamination in Mistral-7b-v0.1, Llama-2-Instruct-70b and Llama-3-70b. All numbers are reported in percentages. Model | Benchmark | Perf. [%] |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---
Llama-2-Instruct-70b | ARC | $61.860\,068$ | $14.680\,000$ | $1.961\,763$ | $-1.014\,571$
| GSM8k | $55.799\,848$ | $58.600\,000$ | $-0.408\,819$ | $-4.420\,310$
| Hellaswag | $85.550\,000$ | $0.410\,000$ | $3.368\,447$ | $1.287\,811$
| MMLU | $56.852\,300$ | $32.910\,000$ | $0.712\,501$ | $-2.097\,160$
Llama-3-70b | ARC | $69.027\,304$ | $0.030\,000$ | $6.609\,663$ | $3.212\,481$
| GSM8k | $81.576\,952$ | $15.450\,000$ | $2.052\,300$ | $-1.486\,972$
| Hellaswag | $86.450\,000$ | $1.010\,000$ | $2.858\,603$ | $0.897\,152$
| MMLU | $76.464\,891$ | $5.760\,000$ | $3.346\,078$ | $-0.212\,983$
Mistral-7b-v0.1 | ARC | $58.959\,044$ | $7.910\,000$ | $2.210\,294$ | $-0.395\,809$
| GSM8k | $39.044\,731$ | $0.150\,000$ | $8.245\,073$ | $4.479\,598$
| Hellaswag | $83.650\,000$ | $0.240\,000$ | $3.135\,526$ | $1.271\,238$
| MMLU | $58.014\,528$ | $15.230\,000$ | $1.880\,507$ | $-1.051\,506$
Table 4: Complete results for all evaluated model families for all tests with result $p<1\%$. All numbers in the table are reported in percentages. Model | Benchmark | Type | Perf. [%] |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Qwen-Instruct-1.5-14b | ARC | B | $56.911\,263$ | $0.710\,000$ | $11.766\,971$ | $5.938\,642$
Qwen-Instruct-1.5-72b | ARC | B | $64.675\,768$ | $0.010\,000$ | $12.593\,272$ | $7.611\,120$
Qwen-Instruct-1.5-110b | ARC | B | $69.453\,925$ | $0.120\,000$ | $9.325\,123$ | $4.469\,376$
Qwen-Instruct-1.5-4b | GSM8k | S | $6.520\,091$ | $<10^{-2}$ | $5.352\,425$ | $4.005\,295$
Qwen-Instruct-1.5-7b | Hellaswag | B | $78.650\,000$ | $0.740\,000$ | $7.460\,459$ | $2.999\,157$
Qwen-Instruct-1.5-14b | Hellaswag | B | $82.150\,000$ | $0.070\,000$ | $6.482\,945$ | $3.737\,166$
Qwen-Instruct-1.5-72b | Hellaswag | B | $86.350\,000$ | $<10^{-2}$ | $8.236\,191$ | $6.052\,811$
Yi-34b | ARC | S | $63.993\,174$ | $0.200\,000$ | $5.003\,248$ | $2.115\,261$
Yi-34b | Hellaswag | S | $86.150\,000$ | $<10^{-2}$ | $6.505\,101$ | $4.399\,920$
Yi-34b | Hellaswag | B | $86.150\,000$ | $0.140\,000$ | $3.957\,500$ | $1.887\,749$
InternLM-2-7b | GSM8k | B | $62.092\,494$ | $0.430\,000$ | $19.271\,184$ | $7.984\,204$
InternLM-2-Math-7b | GSM8k | B | $72.934\,041$ | $<10^{-2}$ | $39.400\,641$ | $27.147\,843$
InternLM-2-7b | Hellaswag | B | $80.100\,000$ | $0.410\,000$ | $6.583\,282$ | $3.189\,791$
InternLM-2-Math-7b | Hellaswag | B | $77.650\,000$ | $0.900\,000$ | $8.552\,391$ | $3.108\,843$
InternLM-2-Math-Base-7b | Hellaswag | B | $79.650\,000$ | $0.400\,000$ | $11.409\,109$ | $5.508\,463$
StableLM-2-12b | ARC | S | $59.470\,990$ | $0.680\,000$ | $4.607\,515$ | $1.593\,007$
StableLM-2-1.6b | GSM8k | B | $18.877\,938$ | $<10^{-2}$ | $16.560\,354$ | $12.953\,915$
StableLM-2-Instruct-1.6b | GSM8k | B | $42.001\,516$ | $<10^{-2}$ | $27.791\,320$ | $19.268\,710$
StableLM-2-Zephyr-3b | GSM8k | B | $51.630\,023$ | $<10^{-2}$ | $48.780\,356$ | $44.338\,575$
StableLM-2-12b | GSM8k | B | $57.998\,484$ | $0.390\,000$ | $17.921\,418$ | $6.961\,541$
StableLM-2-Instruct-12b | GSM8k | B | $68.840\,030$ | $<10^{-2}$ | $32.932\,918$ | $21.094\,482$
StableLM-2-Instruct-12b | Hellaswag | B | $86.250\,000$ | $<10^{-2}$ | $7.037\,107$ | $4.882\,316$
yam-peleg/Experiment26-7b | ARC | B | $72.440\,273$ | $<10^{-2}$ | $22.361\,505$ | $17.279\,298$
| GSM8k | S | $74.526\,156$ | $0.240\,000$ | $7.595\,965$ | $3.193\,949$
| GSM8k | B | $74.526\,156$ | $<10^{-2}$ | $29.688\,649$ | $18.149\,483$
| Hellaswag | B | $88.600\,000$ | $<10^{-2}$ | $13.109\,082$ | $10.273\,189$
BarraHome/Mistroll-7b-v2.2 | ARC | B | $72.525\,597$ | $<10^{-2}$ | $22.207\,655$ | $17.063\,830$
| GSM8k | S | $74.526\,156$ | $0.340\,000$ | $7.342\,518$ | $3.054\,735$
| GSM8k | B | $74.526\,156$ | $<10^{-2}$ | $29.278\,076$ | $17.770\,339$
| Hellaswag | B | $88.600\,000$ | $<10^{-2}$ | $12.954\,633$ | $10.149\,874$
MTSAIR/multi_verse_model | ARC | B | $72.440\,273$ | $<10^{-2}$ | $22.118\,895$ | $16.973\,038$
| GSM8k | S | $74.677\,786$ | $0.530\,000$ | $7.203\,836$ | $2.648\,507$
| GSM8k | B | $74.677\,786$ | $<10^{-2}$ | $29.838\,028$ | $18.425\,013$
| Hellaswag | B | $88.550\,000$ | $<10^{-2}$ | $12.138\,477$ | $9.559\,127$

###  A.2 Results for GSM8k Contaminated Models

We present the complete results for the contaminated models finetuned on the
GSM8k benchmark in Table 5. For a detailed explanation of each of the
settings, we refer to App. C. We do mention here that in the realistic
setting, we only train for 1 epoch, without any few-shot samples in the prompt
and with additional background instruction-tuning data from the OpenOrca
dataset [31].

Table 5: Complete results for the contaminated models finetuned on GSM8k.
Llama-2 is the Llama-2-Instruct-7b model. $\delta$ is the actual effect
measured on the uncontaminated samples. The other values are the estimates
p-values and effects for syntax- and sample-specific contamination. All
numbers in the table are reported in percentages.

Model | Setting | Perf. [%] |  $\delta$ [%] |  $p_{\text{syntax}}$ [%] |  $\hat{\delta}_{\text{syntax}}$ [%] |  $p_{\text{sample}}$ [%] |  $\hat{\delta}_{\text{sample}}$ [%]
---|---|---|---|---|---|---|---
Llama-2 | Default | $92.109\,256$ | $79.381\,984$ | $<10^{-2}$ | $40.350\,861$ | $<10^{-2}$ | $82.774\,420$
| Default, rephrased | $64.643\,399$ | $50.400\,975$ | $99.140\,000$ | $-6.797\,609$ | $<10^{-2}$ | $55.718\,864$
|  learning rate $10^{-4}$ | $73.292\,868$ | $69.959\,535$ | $<10^{-2}$ | $43.604\,272$ | $<10^{-2}$ | $72.175\,238$
|  learning rate $10^{-5}$ | $38.846\,737$ | $15.513\,404$ | $<10^{-2}$ | $12.617\,099$ | $<10^{-2}$ | $19.084\,580$
| Trained for 1 epoch | $25.189\,681$ | $7.916\,954$ | $0.940\,000$ | $5.424\,925$ | $<10^{-2}$ | $11.943\,160$
| Other few-shot samples | $89.529\,590$ | $76.347\,772$ | $<10^{-2}$ | $38.075\,745$ | $<10^{-2}$ | $82.803\,218$
| No few-shot samples | $80.273\,141$ | $65.727\,687$ | $<10^{-2}$ | $37.363\,443$ | $<10^{-2}$ | $71.743\,418$
| Realistic | $69.802\,731$ | $50.711\,822$ | $<10^{-2}$ | $32.379\,137$ | $<10^{-2}$ | $56.994\,606$
| Realistic, rephrased | $40.515\,933$ | $20.667\,448$ | $94.500\,000$ | $-4.466\,549$ | $<10^{-2}$ | $25.654\,875$
Phi-2 | Default | $79.514\,416$ | $36.029\,567$ | $<10^{-2}$ | $14.960\,337$ | $<10^{-2}$ | $41.456\,468$
| Default, rephrased | $69.195\,751$ | $19.953\,327$ | $94.930\,000$ | $-4.068\,651$ | $<10^{-2}$ | $22.314\,252$
|  learning rate $10^{-4}$ | $82.245\,827$ | $62.245\,827$ | $<10^{-2}$ | $33.390\,926$ | $<10^{-2}$ | $68.422\,480$
|  learning rate $10^{-5}$ | $60.091\,047$ | $6.454\,683$ | $39.060\,000$ | $0.635\,514$ | $21.380\,000$ | $2.273\,549$
| Trained for 1 epoch | $55.386\,950$ | $9.023\,314$ | $6.980\,000$ | $3.904\,808$ | $0.050\,000$ | $10.336\,062$
| Other few-shot samples | $81.335\,357$ | $38.759\,599$ | $<10^{-2}$ | $19.951\,072$ | $<10^{-2}$ | $43.563\,054$
| No few-shot samples | $64.188\,164$ | $20.551\,800$ | $0.750\,000$ | $6.484\,225$ | $<10^{-2}$ | $21.609\,772$
| Realistic | $59.787\,557$ | $12.060\,284$ | $4.930\,000$ | $4.295\,801$ | $<10^{-2}$ | $16.449\,730$
| Realistic, rephrased | $60.546\,282$ | $6.303\,858$ | $88.040\,000$ | $-2.731\,049$ | $1.480\,000$ | $6.921\,527$

##  Appendix B Ablation Study via Simulation

To further investigate the performance of ConStat, we conduct an ablation
study using simulations. This approach allows us to test various scenarios and
understand the behavior of ConStat under different conditions without the need
for finetuning or computationally intense evaluations. Furthermore, it helps
in verifying the p-values returned by various tests while avoiding the risk of
tuning our final test to our analysis in Section 4. We first explain the
simulation setup and then present the results.

###  B.1 Simulation and Setup

#### Simulation

In the simulation, samples are modeled as real numbers representing their
complexity. Each sample $x$ in a benchmark $D$ is drawn from a benchmark-
specific distribution $\mathcal{D}$. Therefore, a benchmark $D$ can be
specified by a number $n\in\mathbb{Z}_{>0}$, indicating the number of samples,
and a distribution $\mathcal{D}$, from which the samples are drawn. Given the
benchmarks $(n,\mathcal{D})$ and $(n_{\text{ref}},\mathcal{D}_{\text{ref}})$,
a model is represented as $M=(m,m_{\text{ref}})\in\mathbb{R}^{2}$, where each
number indicates the quality of the model on the respective benchmarks. The
probability that a model $M$ answers a sample $x\in D$ correctly is given by
the formula $\min(1,\exp(-x/m))$. Note that this probability increases as the
quality of the model $m$ grows and decreases as the complexity of the sample
$x$ increases.

If $m_{\text{ref}}<m$ for a given model, the model is contaminated. Reference
models can be drawn from a distribution $\mathcal{M}$ over the real numbers
such that $m_{\text{ref}}=m$ for each reference model. To simulate noise in
the evaluation of models, we can add noise to the quality of the reference
models, resulting in $m_{\text{ref}}\approx m$.

#### Statistical Tests

We compare ConStat against various other statistical tests that one could
construct. First, we include various variants of ConStat. ConStat-No-Sort does
not sort the reference models by their and fits the hardness correction
function $H_{D_{\text{ref}}}$ directly on the scores
$(S_{D_{\text{ref}}}(M_{i}),S_{D}(M_{i}))$. ConStat-No-Random does not include
a random model in the set of reference models. ConStat-No-Bootstrap only
performs bootstrapping over the samples and not over the reference models.

We also include two alternative tests. Mean-Test directly compares performance
on the reference and original benchmarks, considering a model contaminated if
its performance on the original benchmark is significantly higher. Normalized-
Test instead computes the normalized performance with respect to the models by
computing the means $\mu_{D},\mu_{D_{\text{ref}}}$ and standard deviations
$\sigma_{D},\sigma_{D_{\text{ref}}}$ of the reference models on each
benchmark. It then bootstraps the reference models and samples to obtain the
p-value as the probability that the normalized performance
$\sigma_{D}^{-1}(S_{D}(M)-\mu_{D})$ of the model on the original benchmark is
higher than on the reference benchmark. Therefore, Normalized-Test essentially
corrects for first- and second-order distributional differences between
$\mathcal{D}$ and $\mathcal{D}_{\text{ref}}$.

#### Reporting Results

We report results for specific distributions $\mathcal{M},\mathcal{D}$ and
$\mathcal{D}_{\text{ref}}$ for a model $M$ for which we aim to detect possible
contamination. For each choice of distributions, we run 1000 simulations, each
drawing new reference models and benchmarks from the given distributions and
performing the tests described before. This ablation focuses on the
uncontaminated case, where $m=m_{\text{ref}}$, as avoiding false positives is
crucial. In Fig. 3(a), we show an example plot of the resulting CDF of the
returned p-value when $\mathcal{D}=\mathcal{D}_{\text{ref}}$. As expected, the
CDF for each method is very close to the identity line for the uncontaminated
model. Ideally, the curve for uncontaminated models should be as close as
possible to this identity line to ensure reliable p-values. Above the line is
especially problematic, as this would be the cause of false positives.
Furthermore, by swapping the distributions $\mathcal{D}$ and
$\mathcal{D}_{\text{ref}}$, one would obtain a mirror image of the plot. This
means that a CDF that is below the identity line in a given situation, would
be above the identity line in the mirror image, and is therefore also
problematic for the same reason.

We now report results for various scenarios. In each case, we aim to ensure
that a specific test fails and explain why this is the case. In all explored
scenarios, ConStat performs as expected, while the other methods fail. Unless
specified otherwise, we use $20$ reference models and $1000$ samples for each
benchmark, in line with our results presented in Section 4. A full overview of
each parameter setting can be found in Table 6. We always set the quality of
the model under consideration to $(1,1)$.

Table 6: Settings for the simulation scenarios. We use the union of two distributions to indicate the distribution that samples from both distributions with equal probability. The $\sigma$ column indicates the noise added to the reference models, using a normal distribution with mean $0$ and standard deviation $\sigma$. A normal distribution is denoted using the notation $\mathcal{N}(\mu,\sigma)$ where $\sigma$ is the standard deviation. Scenario | $\mathcal{M}$ | $\mathcal{D}$ | $\mathcal{D}_{\text{ref}}$ | $\sigma$
---|---|---|---|---
Different Distributions | $\mathcal{N}(1,0.3)$ | $\mathcal{N}(0.4,0.3)$ | $\mathcal{N}(0.8,0.2)$ | $0$
Non-Linearity | $\mathcal{N}(0.6,0.2)$ | $\mathcal{N}(0.8,0.1)\cup\mathcal{N}(1.4,0.1)$ | $\mathcal{N}(0.3,0.1)\cup\mathcal{N}(1,0.1)$ | $0$
Noise | $\mathcal{N}(0.8,0.1)$ | $\mathcal{N}(1,0.4)$ | $\mathcal{N}(1,0.4)$ | $0.05$
Bootstrapping Models | $\mathcal{N}(0.6,1)$ | $\mathcal{N}(0.8,0.1)\cup\mathcal{N}(1.4,0.1)$ | $\mathcal{N}(0.3,0.1)\cup\mathcal{N}(1,0.1)$ | $0.1$
No Random Model | $\mathcal{N}(4,1)$ | $\mathcal{N}(4,0.2)\cup\mathcal{N}(0.8,0.8)$ | $\mathcal{N}(0.8,0.8)$ | $0.05$

###  B.2 Results

#### Different Distributions

The Mean-Test should fail if the difficulty of one benchmark is different than
the other. We slightly decrease the difficulty of the samples in the original
benchmark to make Mean-Test return false positives, as shown in Fig. 3(b).
Despite $M$ being uncontaminated, the p-values returned by Mean-Test show a
very steep CDF.

#### Non-Linearity

Normalized-Test assumes a linear relationship between performances on
reference and original benchmarks, but non-linear relationships can occur. For
instance, it is non-linear for sample-specific contamination in the GSM8k
benchmark (see Fig. 1). Therefore, we change the benchmark distributions to
ensure a non-linear relationship. The result shown in Fig. 3(c) shows that
Normalized-Test returns a very steep CDF. We note that ConStat-No-Sort also
returns a steep CDF in this particular case.

#### Noise

When reference models do not have the same quality on both benchmarks, noise
is introduced in the signal that the test receives. Our theoretical analysis
in Section 3 corrects for this noise by sorting the reference models by
performance on each benchmark. ConStat-No-Sort is more susceptible to this
noise. We showcase this for an uncontaminated model in Fig. 3(d) by keeping
$\mathcal{D}=\mathcal{D}_{\text{ref}}$, but now adding a small amount of noise
to the reference models. ConStat-No-Sort returns a steep CDF and should not be
used in practice due to the noisy nature of real-world scenarios.

#### Bootstrapping Models

Bootstrapping over reference models is necessary for reliable p-values.
Without bootstrapping, the test would rely on the specific instantiation of
the reference models, leading to p-values that are either too certain, always
returning $0$ or $1$. We repeat the non-linear scenario with added noise to
the reference models and a wider distribution over them. As shown in Fig.
3(e), ConStat-No-Bootstrap returns a CDF that is very steep at either edge.

#### No Random Model

Adding a random model to the reference models in ConStat provides further
regularization. The effect of this addition only becomes apparent when we use
fewer reference models in a non-linear scenario. In such cases, all reference
models are relatively close together and the smoothing spline overfits to this
local part of the curve. We demonstrate this in Fig. 3(f), using only five
reference models and a non-linear relationship between the benchmarks.
ConStat-No-Random shows a rather steep CDF in this scenario.

We conclude that ConStat is robust to various scenarios and provides reliable
p-values in all cases. The other tests fail in the scenarios we have
presented, highlighting the importance of the design choices made in ConStat.

((a)) A simple scenario where all tests should return a CDF close to the
identity line.

((b)) A scenario where we make the distributions of the benchmarks different.

((c)) The relationship between performances on the reference and original
benchmarks is non-linear.

((d)) The reference models are noisy and the relationship between the
benchmarks is linear.

((e)) The reference models are noisy and the relationship between the
benchmarks is non-linear.

((f)) A small number of reference models and a non-linear relationship between
the benchmarks. ConStat-No-Bootstrap and ConStat are the same in this case.

Figure 3: CDF of various statistical tests for uncontaminated models in
different scenarios.

##  Appendix C Experimental Details

We describe the full details for the experiments presented in Section 4. This
includes the preprocessing stage of the benchmarks, the data generation
process, the fine-tuning of the models on the benchmarks, and the evaluation
of the models, including the reference models used. Additionally, we provide
details on the computational resources necessary to run the experiments.
Licensing information for all assets used in the experiments is provided in
App. D.

#### Preprocessing

We select four benchmarks for our experiments, ARC [15], GSM8k [16], Hellaswag
[53], and MMLU [26]. Due to the large size of MMLU, we first select a subset
of the topics from which it consists. Specifically, we select the following
topics: Abstract Algebra, Anatomy, Astronomy, Business Ethics, Clinical
Knowledge, College Biology, College Chemistry, College Computer Science,
College Mathematics, College Medicine, College Physics, Computer Security,
Conceptual Physics, Econometrics, and Electrical Engineering. We randomly
select 2000 samples from each of the benchmarks. These samples were then split
into two equally-sized sets, one of which was used for contaminating the fine-
tuned models.

For the chosen reference benchmarks we limit the number of samples to 2000.

#### Data Generation

For each benchmark, we generate a rephrased version of the benchmark and a
synthetic benchmark. For both these purposes, we use GPT-4-Turbo [35].
Specifically, for the rephrased benchmarks, we write a system prompt asking
the model to rephrase the input (including options for multiple-choice
benchmarks) of a given sample. Furthermore, we use a different system prompt
to generate a rephrased benchmark that rephrases both input and output. This
second rephrased benchmark is used to finetune the models that are trained on
rephrased data for Section 4.2. By using separate prompts for training and
evaluation, we ensured that the evaluation did not occur on the same data as
the training.

For the synthetic benchmarks, we write a system prompt that asks to generate
new synthetic samples for the benchmark. To obtain faithful synthetic samples,
we use few-shotting where the model is given several examples of the
benchmark. By placing these generated samples in the "assistant" field of the
chat model and changing the given few-shot examples for each sample, we
ensured both faithful and diverse samples. We generated 1000 samples for each
benchmark.

Post-processing was then applied to the synthetically generated benchmark
samples. First, duplicates within the synthetic samples were removed by
searching for high 1-gram overlap ratios between two samples. Second, we
removed samples with a high 1-gram overlap ratio with the original benchmark
samples, ensuring the synthetic samples were not too similar to the originals.
Specifically, for the GSM8k benchmark, we found that some generated samples
did not result in an integer answer, or the model used a rounding operation.
These samples were removed by checking if the answer was an integer and
ensuring no rounding was involved.

The system prompts used for the rephrased benchmarks and synthetic benchmarks
are available in the code repository.

#### Finetuning

We explain the finetuning process for the Phi-2 and Llama-2-Instruct-7b models
that were used in Section 4.2. We use the Hugging Face Transformers library
[47] for the finetuning process.

Specifically, we applied full finetuning with batch size 16 and the Adam
optimizer on different datasets and using different hyperparameters. We use
the following default hyperparameters:

  * •

A learning rate of $5\cdot 10^{-5}$.

  * •

The dataset on which we train is the contaminatable part of a given benchmark.

  * •

We train for 5 epochs.

  * •

The prompt includes the exact few-shot samples used for evaluation.

We then train $8$ other models where we always change specific parameters in
this default setting. Specifically, we train models that diverge from the
default setting in the following ways:

  1. 1.

Instead of training with the exact samples from the benchmark, we train on the
rephrased benchmark.

  2. 2.

We change the learning rate to $10^{-5}$.

  3. 3.

We change the learning rate to $10^{-4}$.

  4. 4.

We only train for $1$ epoch.

  5. 5.

We train without any few-shot samples in the prompt.

  6. 6.

We train with a random set of few-shot samples instead of the few-shot samples
from the benchmark.

  7. 7.

We do not include any few-shot samples in the prompt, include additional
background instruction-tuning data from the OpenOrca dataset [31], and only
train for $1$ epoch.

  8. 8.

We do the same as in the previous setting, with the additional change that we
train on the rephrased benchmark instead of the actual one.

By including such a wide range of possible settings, we ensure that we cover a
wide range of possible contamination effects. As can be seen in Fig. 2, the
resulting models indeed show varying levels of contamination from $0\%$ up to
$80\%$.

#### Reference Models

The following models were used as reference models in our experiments: Phi-2,
Phi-3, Llama-2-7b, Llama-2-Instruct-7b, Llama-2-13b, Llama-2-Instruct-13b,
Llama-2-Instruct-70b, Llama-3-8b, Llama-3-Instruct-8b, Llama-3-70b,
Llama-3-Instruct-70b, Mistral-7b-v0.1, Mistral-Instruct-7b-v0.1, Mistral-
Instruct-7b-v0.2, Mixtral-Instruct-8x7b, Mixtral-Instruct-8x22b,
Falcon-7b,Falcon-Instruct-7b, Gemma-1.1-7b, Gemma-1.1-Instruct-7b, OLMo-
Instruct-7b. As discussed in Section 4.3, we removed Mistral-7b-v0.1 from the
reference models after a contamination analysis.

#### Evaluation

We evaluate the models with v0.4.1 of the LM Evaluation Harness [20]. We use
$5$-shot evaluation for all models and provide the custom fork of the
evaluation harness to allow for the evaluation on all the synthetic and
rephrased benchmarks in our code repository.

#### Compute

We spent around $300$ USD on the OpenAI API to generate all benchmarks.
Furthermore, we used a single Nvidia H100 GPU for around 1 month to finetune
and evaluate all models. Finally, for models that were too large to fit on a
single GPU, we used the Together API to run inference. We spent an additional
$263$ USD on this platform.

##  Appendix D Licensing Information

We include the license for all models, benchmarks and other assets used in
this paper in Table 7.

Table 7: Table with assets used, description of their use and the license under which they are distributed. Sections are split by the type of asset: benchmarks, code repositories and then models. Asset |  Description & Use |  License Name
---|---|---
MMLU [26] |  General-purpose benchmark used for evaluation |  MIT License
Hellaswag [53] |  General-purpose benchmark used for evaluation |  MIT License
GSM8k [16] |  General-purpose benchmark used for evaluation |  MIT License
ARC-Challenge [15] |  General-purpose benchmark used for evaluation |  CC-BY-SA-4.0
OpenOrca [31] |  Instruction-tuning dataset used in finetuning process |  MIT License
LM Evaluation Harness [20] |  Framework used to perform evaluations |  MIT License
[39] |  Repository to run the [39] baseline |  Not Specified
Scipy [43] |  Adapted code for smoothing spline fitting |  BSD 3-Clause License
Llama-2 [42] |  Includes all Llama-2 models, were evaluated for contamination and used for finetuning |  Llama 2 Community License Agreement
Llama-3 [2] |  Includes all Llama-3 models, were evaluated for contamination |  Llama 3 Community License Agreement
Falcon [3] |  Includes all Falcon models, were evaluated for contamination |  Apache 2.0 License
Gemma-1.1 [21] |  Includes all Gemma-1.1 models, were evaluated for contamination |  Gemma Terms of Use
yam-peleg/Experiment26-7b |  Was evaluated for contamination |  Apache 2.0 License
BarraHome/Mistroll-7b-v2.2 |  Was evaluated for contamination |  MIT License
MTSAIR/multi_verse_model |  Was evaluated for contamination |  Apache 2.0 License
Mistral [28] |  Includes all Mistral models, were evaluated for contamination |  Apache 2.0 License
Phi-2 [27] |  Was evaluated for contamination and used for finetuning |  MIT License
Phi-3 [1] |  Was evaluated for contamination. |  MIT License
Qwen-1.5 [6] |  Includes all Qwen-1.5 models, were evaluated for contamination |  Tongyi Qianwen License Agreement
StableLM-2 [8] |  Includes all StableLM-2 models, were evaluated for contamination |  Stability Ai Non-Commercial Research Community License Agreement
InternLM-2 [11] |  Includes all InternLM-2 models, were evaluated for contamination |  Apache 2.0 License
OLMo [25] |  Includes all OLMo models, were evaluated for contamination |  Apache 2.0 License
Yi [52] |  Includes all Yi models, were evaluated for contamination |  Yi Series Models Community License Agreement
GPT-4-Turbo |  Used to generate synthetic and rephrased benchmarks |  OpenAI Terms of Use

##  Appendix E All Test Results

We present all results for each performed test in this section. Tables Table
8-Table 23 contain these results, grouped by model family.

Table 8: Contamination results for yam-peleg/Experiment26-7b, MTSAIR/multi_verse_model, BarraHome/Mistroll-7b-v2.2. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
BarraHome/Mistroll-7b-v2.2 | ARC | $72.525\,597$ | B | $<10^{-2}$ | $22.207\,655$ | $17.063\,830$
|  |  | S | $55.900\,000\,000\,000\,006$ | $-0.220\,593$ | $-4.883\,328$
|  |  | Y | $88.149\,999\,999\,999\,99$ | $-3.086\,816$ | $-7.256\,855$
| GSM8k | $74.526\,156$ | B | $<10^{-2}$ | $29.278\,076$ | $17.770\,339$
|  |  | S | $0.339\,999\,999\,999\,999\,97$ | $7.342\,518$ | $3.054\,735$
|  |  | Y | $34.0$ | $0.785\,755$ | $-2.492\,201$
| Hellaswag | $88.600\,000$ | B | $<10^{-2}$ | $12.954\,633$ | $10.149\,874$
|  |  | S | $96.48$ | $-3.245\,091$ | $-6.006\,571$
|  |  | Y | $12.879\,999\,999\,999\,999$ | $1.353\,101$ | $-0.566\,034$
| MMLU | $58.886\,199$ | S | $35.65$ | $0.714\,669$ | $-2.413\,310$
|  |  | Y | $85.66$ | $-1.833\,037$ | $-4.616\,790$
MTSAIR/multi_verse_model | ARC | $72.440\,273$ | B | $<10^{-2}$ | $22.118\,895$ | $16.973\,038$
|  |  | S | $51.070\,000\,000\,000\,01$ | $0.126\,554$ | $-4.490\,522$
|  |  | Y | $87.29$ | $-2.907\,406$ | $-6.986\,894$
| GSM8k | $74.677\,786$ | B | $<10^{-2}$ | $29.838\,028$ | $18.425\,013$
|  |  | S | $0.53$ | $7.203\,836$ | $2.648\,507$
|  |  | Y | $47.260\,000\,000\,000\,005$ | $0.123\,431$ | $-3.015\,931$
| Hellaswag | $88.550\,000$ | B | $<10^{-2}$ | $12.138\,477$ | $9.559\,127$
|  |  | S | $96.399\,999\,999\,999\,99$ | $-3.187\,084$ | $-5.919\,225$
|  |  | Y | $10.12$ | $1.499\,898$ | $-0.398\,864$
| MMLU | $58.983\,051$ | S | $33.989\,999\,999\,999\,995$ | $0.819\,701$ | $-2.332\,535$
|  |  | Y | $81.479\,999\,999\,999\,99$ | $-1.516\,036$ | $-4.300\,854$
yam-peleg/Experiment26-7b | ARC | $72.440\,273$ | B | $<10^{-2}$ | $22.361\,505$ | $17.279\,298$
|  |  | S | $60.099\,999\,999\,999\,994$ | $-0.529\,206$ | $-5.262\,421$
|  |  | Y | $90.97$ | $-3.806\,369$ | $-8.085\,049$
| GSM8k | $74.526\,156$ | B | $<10^{-2}$ | $29.688\,649$ | $18.149\,483$
|  |  | S | $0.24$ | $7.595\,965$ | $3.193\,949$
|  |  | Y | $14.74$ | $2.066\,568$ | $-1.191\,276$
| Hellaswag | $88.600\,000$ | B | $<10^{-2}$ | $13.109\,082$ | $10.273\,189$
|  |  | S | $96.52$ | $-3.245\,071$ | $-6.004\,187$
|  |  | Y | $13.350\,000\,000\,000\,001$ | $1.355\,751$ | $-0.591\,345$
| MMLU | $59.031\,477$ | S | $23.810\,000\,000\,000\,002$ | $1.364\,031$ | $-1.787\,940$
|  |  | Y | $78.91$ | $-1.362\,910$ | $-4.166\,475$
Table 9: Contamination results for Qwen-Instruct-1.5-4b, Qwen-Instruct-1.5-7b, Qwen-Instruct-1.5-1.8b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Qwen-Instruct-1.5-1.8b | ARC | $38.395\,904$ | B | $97.39$ | $-6.467\,902$ | $-12.365\,814$
|  |  | S | $65.23$ | $-1.169\,238$ | $-6.556\,288$
|  |  | Y | $75.55$ | $-1.381\,122$ | $-4.723\,866$
| GSM8k | $31.159\,970$ | B | $16.59$ | $6.434\,752$ | $-4.075\,318$
|  |  | S | $26.479\,999\,999\,999\,997$ | $1.457\,019$ | $-2.380\,261$
|  |  | Y | $5.62$ | $3.425\,684$ | $-0.128\,473$
| Hellaswag | $60.900\,000$ | B | $100.0$ | $-9.291\,757$ | $-13.982\,064$
|  |  | S | $99.729\,999\,999\,999\,99$ | $-7.343\,111$ | $-11.269\,887$
|  |  | Y | $82.43$ | $-2.445\,059$ | $-7.911\,157$
| MMLU | $41.694\,915$ | S | $34.599\,999\,999\,999\,994$ | $0.557\,081$ | $-2.433\,621$
|  |  | Y | $32.23$ | $0.794\,008$ | $-2.151\,228$
Qwen-Instruct-1.5-4b | ARC | $42.064\,846$ | B | $30.330\,000\,000\,000\,002$ | $6.033\,683$ | $-22.110\,356$
|  |  | S | $32.46$ | $0.791\,484$ | $-3.958\,037$
|  |  | Y | $86.9$ | $-2.095\,630$ | $-5.244\,428$
| GSM8k | $6.520\,091$ | B | $99.82$ | $-25.427\,456$ | $-36.934\,330$
|  |  | S | $<10^{-2}$ | $5.352\,425$ | $4.005\,295$
|  |  | Y | $11.87$ | $1.592\,154$ | $-0.614\,462$
| Hellaswag | $69.350\,000$ | B | $93.97$ | $-3.530\,716$ | $-7.063\,561$
|  |  | S | $66.77$ | $-0.846\,198$ | $-4.222\,447$
|  |  | Y | $26.14$ | $1.360\,052$ | $-3.003\,603$
| MMLU | $50.363\,196$ | S | $45.01$ | $0.246\,671$ | $-2.873\,510$
|  |  | Y | $88.31$ | $-2.143\,078$ | $-5.086\,228$
Qwen-Instruct-1.5-7b | ARC | $55.119\,454$ | B | $3.05$ | $14.285\,379$ | $2.244\,218$
|  |  | S | $59.77$ | $-0.469\,250$ | $-3.606\,222$
|  |  | Y | $57.67$ | $-0.331\,862$ | $-3.284\,673$
| GSM8k | $55.117\,513$ | B | $59.199\,999\,999\,999\,996$ | $-1.383\,031$ | $-11.909\,238$
|  |  | S | $9.790\,000\,000\,000\,001$ | $3.728\,790$ | $-1.093\,002$
|  |  | Y | $29.53$ | $1.160\,183$ | $-2.538\,211$
| Hellaswag | $78.650\,000$ | B | $0.74$ | $7.460\,459$ | $2.999\,157$
|  |  | S | $90.61$ | $-1.873\,664$ | $-4.176\,442$
|  |  | Y | $15.2$ | $1.741\,057$ | $-1.158\,455$
| MMLU | $57.820\,823$ | S | $69.8$ | $-1.022\,455$ | $-4.366\,419$
|  |  | Y | $50.88$ | $-0.019\,586$ | $-3.036\,964$
Table 10: Contamination results for Qwen-Instruct-1.5-14b, Qwen-Instruct-1.5-72b, Qwen-Instruct-1.5-110b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Qwen-Instruct-1.5-110b | ARC | $69.453\,925$ | B | $0.12$ | $9.325\,123$ | $4.469\,376$
|  |  | S | $54.0$ | $-0.202\,895$ | $-4.134\,481$
|  |  | Y | $56.379\,999\,999\,999\,995$ | $-0.254\,900$ | $-3.289\,120$
| GSM8k | $82.107\,657$ | B | $79.77$ | $-2.148\,756$ | $-7.636\,509$
|  |  | S | $20.01$ | $2.047\,895$ | $-1.941\,723$
|  |  | Y | $7.08$ | $2.803\,834$ | $-0.350\,747$
| Hellaswag | $84.250\,000$ | B | $29.03$ | $0.694\,719$ | $-1.475\,117$
|  |  | S | $93.86$ | $-2.307\,738$ | $-4.705\,950$
|  |  | Y | $25.419\,999\,999\,999\,998$ | $0.934\,851$ | $-1.504\,779$
| MMLU | $73.995\,157$ | S | $65.01$ | $-0.851\,254$ | $-4.874\,204$
|  |  | Y | $67.92$ | $-0.723\,015$ | $-3.400\,211$
Qwen-Instruct-1.5-14b | ARC | $56.911\,263$ | B | $0.710\,000\,000\,000\,000\,1$ | $11.766\,971$ | $5.938\,642$
|  |  | S | $88.649\,999\,999\,999\,99$ | $-2.496\,819$ | $-5.716\,544$
|  |  | Y | $32.93$ | $0.769\,973$ | $-2.112\,803$
| GSM8k | $68.385\,140$ | B | $5.12$ | $10.284\,514$ | $-0.043\,177$
|  |  | S | $1.13$ | $6.365\,559$ | $1.890\,827$
|  |  | Y | $75.33$ | $-1.448\,795$ | $-4.943\,067$
| Hellaswag | $82.150\,000$ | B | $0.069\,999\,999\,999\,999\,99$ | $6.482\,945$ | $3.737\,166$
|  |  | S | $98.89$ | $-3.144\,877$ | $-5.394\,183$
|  |  | Y | $3.409\,999\,999\,999\,999\,7$ | $2.606\,891$ | $0.271\,288$
| MMLU | $64.939\,467$ | S | $70.020\,000\,000\,000\,01$ | $-1.122\,665$ | $-4.755\,661$
|  |  | Y | $34.23$ | $0.667\,766$ | $-2.020\,471$
Qwen-Instruct-1.5-72b | ARC | $64.675\,768$ | B | $0.01$ | $12.593\,272$ | $7.611\,120$
|  |  | S | $70.960\,000\,000\,000\,01$ | $-1.340\,009$ | $-5.416\,551$
|  |  | Y | $6.069\,999\,999\,999\,999$ | $2.769\,723$ | $-0.170\,542$
| GSM8k | $79.454\,132$ | B | $5.76$ | $9.737\,556$ | $-0.423\,093$
|  |  | S | $38.33$ | $0.772\,441$ | $-3.412\,319$
|  |  | Y | $98.9$ | $-4.386\,326$ | $-7.540\,562$
| Hellaswag | $86.350\,000$ | B | $<10^{-2}$ | $8.236\,191$ | $6.052\,811$
|  |  | S | $14.24$ | $1.480\,705$ | $-0.758\,134$
|  |  | Y | $27.24$ | $0.761\,483$ | $-1.306\,173$
| MMLU | $74.285\,714$ | S | $11.91$ | $2.645\,919$ | $-1.180\,731$
|  |  | Y | $22.85$ | $1.226\,977$ | $-1.481\,881$
Table 11: Contamination results for OLMo-Instruct-7b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
OLMo-Instruct-7b | ARC | $46.075\,085$ | B | $22.68$ | $7.219\,895$ | $-11.302\,879$
|  |  | S | $45.29$ | $0.158\,959$ | $-2.928\,931$
|  |  | Y | $53.61$ | $-0.100\,325$ | $-3.097\,662$
| GSM8k | $11.751\,327$ | B | $<10^{-2}$ | $8.863\,050$ | $4.989\,178$
|  |  | S | $7.31$ | $2.135\,081$ | $-0.331\,943$
|  |  | Y | $57.97$ | $-0.247\,556$ | $-2.516\,277$
| Hellaswag | $79.950\,000$ | B | $2.989\,999\,999\,999\,999\,8$ | $2.625\,743$ | $0.412\,603$
|  |  | S | $5.04$ | $2.380\,775$ | $-0.003\,295$
|  |  | Y | $17.9$ | $1.141\,475$ | $-1.077\,468$
| MMLU | $41.937\,046$ | S | $70.97$ | $-0.686\,242$ | $-2.884\,344$
|  |  | Y | $48.8$ | $0.032\,109$ | $-2.614\,956$
Table 12: Contamination results for Gemma-1.1-Instruct-2b, Gemma-1.1-Instruct-7b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Gemma-1.1-Instruct-2b | ARC | $44.709\,898$ | B | $99.47$ | $-7.115\,692$ | $-11.794\,727$
|  |  | S | $26.13$ | $0.692\,510$ | $-2.884\,186$
|  |  | Y | $73.27$ | $-0.956\,181$ | $-3.508\,356$
| GSM8k | $10.614\,102$ | B | $100.0$ | $-28.165\,235$ | $-38.869\,715$
|  |  | S | $90.869\,999\,999\,999\,99$ | $-1.797\,054$ | $-3.670\,601$
|  |  | Y | $95.11$ | $-2.430\,760$ | $-4.717\,083$
| Hellaswag | $63.250\,000$ | B | $89.68$ | $-4.468\,643$ | $-9.009\,716$
|  |  | S | $87.14$ | $-2.469\,212$ | $-5.067\,731$
|  |  | Y | $21.240\,000\,000\,000\,002$ | $0.644\,648$ | $-4.853\,043$
| MMLU | $35.641\,646$ | S | $74.14$ | $-0.928\,311$ | $-3.744\,235$
|  |  | Y | $85.61$ | $-1.750\,179$ | $-4.169\,546$
Gemma-1.1-Instruct-7b | ARC | $58.020\,478$ | B | $9.31$ | $3.642\,799$ | $-1.016\,221$
|  |  | S | $10.48$ | $2.128\,186$ | $-0.742\,597$
|  |  | Y | $44.06$ | $0.210\,721$ | $-2.436\,483$
| GSM8k | $50.796\,058$ | B | $98.63$ | $-13.665\,270$ | $-23.140\,873$
|  |  | S | $81.05$ | $-2.227\,299$ | $-6.103\,946$
|  |  | Y | $64.59$ | $-0.721\,559$ | $-4.004\,658$
| Hellaswag | $76.850\,000$ | B | $1.79$ | $5.787\,380$ | $1.586\,747$
|  |  | S | $3.209\,999\,999\,999\,999\,5$ | $3.175\,280$ | $0.395\,412$
|  |  | Y | $13.919\,999\,999\,999\,998$ | $1.763\,207$ | $-1.162\,466$
| MMLU | $53.704\,600$ | S | $58.46$ | $-0.312\,841$ | $-3.056\,708$
|  |  | Y | $90.68$ | $-2.090\,195$ | $-4.584\,779$
Table 13: Contamination results for InternLM-2-7b, InternLM-2-Math-7b, InternLM-2-Math-Base-7b, InternLM-2-1.8b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
InternLM-2-1.8b | ARC | $39.931\,741$ | B | $100.0$ | $-19.045\,015$ | $-23.945\,574$
|  |  | S | $85.399\,999\,999\,999\,99$ | $-2.292\,073$ | $-6.760\,078$
|  |  | Y | $75.13$ | $-1.276\,488$ | $-4.516\,899$
| GSM8k | $24.033\,359$ | B | $5.93$ | $7.983\,371$ | $-0.366\,198$
|  |  | S | $19.37$ | $1.640\,436$ | $-1.615\,638$
|  |  | Y | $91.3$ | $-2.632\,158$ | $-5.830\,702$
| Hellaswag | $63.050\,000$ | B | $98.31$ | $-6.843\,709$ | $-11.651\,523$
|  |  | S | $43.03$ | $0.031\,730$ | $-5.803\,185$
|  |  | Y | $40.35$ | $0.329\,421$ | $-5.289\,282$
| MMLU | $41.259\,080$ | S | $82.17$ | $-1.544\,328$ | $-4.351\,945$
|  |  | Y | $52.23$ | $-0.134\,405$ | $-3.124\,858$
InternLM-2-7b | ARC | $55.546\,075$ | B | $100.0$ | $-13.677\,310$ | $-18.744\,862$
|  |  | S | $3.51$ | $3.442\,550$ | $0.344\,337$
|  |  | Y | $27.51$ | $0.915\,388$ | $-1.793\,602$
| GSM8k | $62.092\,494$ | B | $0.43$ | $19.271\,184$ | $7.984\,204$
|  |  | S | $11.63$ | $3.167\,701$ | $-1.210\,341$
|  |  | Y | $84.26$ | $-2.104\,149$ | $-5.575\,042$
| Hellaswag | $80.100\,000$ | B | $0.410\,000\,000\,000\,000\,03$ | $6.583\,282$ | $3.189\,791$
|  |  | S | $2.64$ | $2.732\,880$ | $0.462\,305$
|  |  | Y | $30.020\,000\,000\,000\,003$ | $0.645\,746$ | $-1.527\,545$
| MMLU | $57.772\,397$ | S | $33.660\,000\,000\,000\,004$ | $0.796\,484$ | $-2.383\,621$
|  |  | Y | $9.9$ | $2.198\,671$ | $-0.620\,987$
InternLM-2-Math-7b | ARC | $52.815\,700$ | B | $84.86$ | $-3.138\,863$ | $-8.283\,806$
|  |  | S | $41.52$ | $0.414\,210$ | $-3.090\,836$
|  |  | Y | $21.279\,999\,999\,999\,998$ | $1.519\,658$ | $-1.610\,429$
| GSM8k | $72.934\,041$ | B | $<10^{-2}$ | $39.400\,641$ | $27.147\,843$
|  |  | S | $64.34$ | $-0.892\,533$ | $-4.871\,255$
|  |  | Y | $71.92$ | $-1.155\,018$ | $-4.357\,915$
| Hellaswag | $77.650\,000$ | B | $0.899\,999\,999\,999\,999\,9$ | $8.552\,391$ | $3.108\,843$
|  |  | S | $5.57$ | $2.796\,742$ | $-0.107\,616$
|  |  | Y | $33.900\,000\,000\,000\,006$ | $0.640\,806$ | $-2.482\,492$
| MMLU | $57.481\,840$ | S | $33.57$ | $0.860\,864$ | $-2.487\,470$
|  |  | Y | $53.43$ | $-0.141\,637$ | $-3.151\,557$
InternLM-2-Math-Base-7b | ARC | $56.228\,669$ | B | $64.31$ | $-0.986\,419$ | $-5.500\,410$
|  |  | S | $21.6$ | $1.488\,184$ | $-1.678\,398$
|  |  | Y | $38.46$ | $0.517\,954$ | $-2.324\,170$
| GSM8k | $35.633\,055$ | B | $83.69$ | $-7.166\,859$ | $-18.749\,159$
|  |  | S | $1.48$ | $5.447\,302$ | $1.568\,005$
|  |  | Y | $14.23$ | $2.355\,058$ | $-1.248\,035$
| Hellaswag | $79.650\,000$ | B | $0.4$ | $11.409\,109$ | $5.508\,463$
|  |  | S | $4.63$ | $2.408\,765$ | $0.054\,354$
|  |  | Y | $35.4$ | $0.479\,840$ | $-1.800\,476$
| MMLU | $52.832\,930$ | S | $48.63$ | $0.077\,939$ | $-3.155\,870$
|  |  | Y | $11.600\,000\,000\,000\,001$ | $2.359\,304$ | $-0.932\,011$
Table 14: Contamination results for Llama-2-Instruct-7b, Llama-2-7b, Llama-2-Instruct-13b, Llama-2-13b, Llama-2-Instruct-70b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Llama-2-13b | ARC | $56.228\,669$ | B | $98.26$ | $-5.455\,626$ | $-9.430\,425$
|  |  | S | $13.56$ | $1.622\,227$ | $-0.919\,786$
|  |  | Y | $48.39$ | $0.072\,851$ | $-2.021\,221$
| GSM8k | $23.805\,914$ | B | $72.02$ | $-3.408\,109$ | $-13.580\,955$
|  |  | S | $13.139\,999\,999\,999\,999$ | $1.743\,083$ | $-1.244\,190$
|  |  | Y | $78.51$ | $-1.328\,567$ | $-4.117\,221$
| Hellaswag | $82.350\,000$ | B | $97.330\,000\,000\,000\,01$ | $-2.007\,131$ | $-3.689\,617$
|  |  | S | $2.07$ | $2.379\,159$ | $0.506\,027$
|  |  | Y | $51.41$ | $-0.032\,123$ | $-1.491\,179$
| MMLU | $48.523\,002$ | S | $73.33$ | $-0.883\,419$ | $-3.214\,572$
|  |  | Y | $70.56$ | $-0.759\,089$ | $-3.140\,807$
Llama-2-7b | ARC | $52.559\,727$ | B | $59.440\,000\,000\,000\,005$ | $-0.663\,740$ | $-5.099\,250$
|  |  | S | $4.07$ | $2.890\,542$ | $0.208\,219$
|  |  | Y | $18.310\,000\,000\,000\,002$ | $1.142\,673$ | $-1.057\,052$
| GSM8k | $13.419\,257$ | B | $38.75$ | $1.604\,078$ | $-5.562\,665$
|  |  | S | $35.13$ | $0.370\,422$ | $-1.680\,401$
|  |  | Y | $53.37$ | $-0.090\,275$ | $-2.375\,855$
| Hellaswag | $78.950\,000$ | B | $98.22$ | $-2.420\,794$ | $-4.347\,052$
|  |  | S | $5.41$ | $2.026\,970$ | $-0.053\,175$
|  |  | Y | $44.09$ | $0.132\,183$ | $-1.655\,578$
| MMLU | $41.307\,506$ | S | $7.68$ | $2.910\,564$ | $-0.537\,253$
|  |  | Y | $14.99$ | $1.539\,103$ | $-1.140\,226$
Llama-2-Instruct-13b | ARC | $56.569\,966$ | B | $3.699\,999\,999\,999\,999\,7$ | $4.976\,786$ | $0.485\,478$
|  |  | S | $72.99$ | $-0.894\,543$ | $-3.456\,376$
|  |  | Y | $78.67$ | $-1.154\,683$ | $-3.559\,421$
| GSM8k | $36.694\,466$ | B | $35.55$ | $2.699\,000$ | $-8.222\,836$
|  |  | S | $35.35$ | $0.615\,276$ | $-2.920\,552$
|  |  | Y | $35.94$ | $0.664\,172$ | $-2.460\,522$
| Hellaswag | $82.350\,000$ | B | $77.82$ | $-0.851\,431$ | $-2.753\,465$
|  |  | S | $18.86$ | $1.046\,358$ | $-0.950\,238$
|  |  | Y | $10.86$ | $1.295\,524$ | $-0.465\,631$
| MMLU | $46.828\,087$ | S | $80.089\,999\,999\,999\,99$ | $-1.258\,183$ | $-3.546\,979$
|  |  | Y | $86.45$ | $-1.722\,962$ | $-4.077\,878$
Llama-2-Instruct-70b | ARC | $61.860\,068$ | B | $28.49$ | $1.342\,766$ | $-2.695\,303$
|  |  | S | $14.680\,000\,000\,000\,001$ | $1.961\,763$ | $-1.014\,571$
|  |  | Y | $87.74$ | $-1.772\,814$ | $-4.207\,193$
| GSM8k | $55.799\,848$ | B | $7.23$ | $9.180\,231$ | $-1.111\,410$
|  |  | S | $58.599\,999\,999\,999\,994$ | $-0.408\,819$ | $-4.420\,310$
|  |  | Y | $57.699\,999\,999\,999\,996$ | $-0.327\,639$ | $-3.552\,515$
| Hellaswag | $85.550\,000$ | B | $6.439\,999\,999\,999\,999\,5$ | $1.640\,463$ | $-0.151\,107$
|  |  | S | $0.410\,000\,000\,000\,000\,03$ | $3.368\,447$ | $1.287\,811$
|  |  | Y | $71.66$ | $-0.520\,879$ | $-2.108\,929$
| MMLU | $56.852\,300$ | S | $32.910\,000\,000\,000\,004$ | $0.712\,501$ | $-2.097\,160$
|  |  | Y | $43.26$ | $0.245\,126$ | $-2.251\,640$
Llama-2-Instruct-7b | ARC | $51.023\,891$ | B | $4.74$ | $5.012\,763$ | $0.128\,220$
|  |  | S | $82.83$ | $-1.519\,363$ | $-4.050\,891$
|  |  | Y | $76.259\,999\,999\,999\,99$ | $-0.957\,215$ | $-3.190\,894$
| GSM8k | $22.744\,503$ | B | $16.17$ | $5.400\,085$ | $-2.917\,097$
|  |  | S | $68.05$ | $-0.593\,130$ | $-2.996\,870$
|  |  | Y | $7.9$ | $2.353\,054$ | $-0.413\,721$
| Hellaswag | $78.100\,000$ | B | $91.52$ | $-1.612\,460$ | $-3.524\,286$
|  |  | S | $93.08$ | $-1.879\,637$ | $-3.885\,386$
|  |  | Y | $53.49$ | $-0.120\,079$ | $-2.132\,074$
| MMLU | $43.244\,552$ | S | $86.64$ | $-1.666\,427$ | $-3.781\,495$
|  |  | Y | $43.76$ | $0.200\,998$ | $-2.220\,837$
Table 15: Contamination results for Llama-3-Instruct-70b, Llama-3-70b, Llama-3-8b, Llama-3-Instruct-8b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Llama-3-70b | ARC | $69.027\,304$ | B | $46.06$ | $0.261\,692$ | $-4.051\,766$
|  |  | S | $0.03$ | $6.609\,663$ | $3.212\,481$
|  |  | Y | $1.95$ | $3.352\,051$ | $0.729\,430$
| GSM8k | $81.576\,952$ | B | $92.86$ | $-5.033\,716$ | $-9.550\,701$
|  |  | S | $15.45$ | $2.052\,300$ | $-1.486\,972$
|  |  | Y | $23.24$ | $1.133\,927$ | $-1.603\,168$
| Hellaswag | $86.450\,000$ | B | $77.74$ | $-0.743\,796$ | $-2.430\,135$
|  |  | S | $1.01$ | $2.858\,603$ | $0.897\,152$
|  |  | Y | $94.19$ | $-2.011\,582$ | $-3.815\,394$
| MMLU | $76.464\,891$ | S | $5.76$ | $3.346\,078$ | $-0.212\,983$
|  |  | Y | $55.02$ | $-0.118\,487$ | $-2.212\,377$
Llama-3-8b | ARC | $56.911\,263$ | B | $99.41$ | $-6.456\,141$ | $-10.301\,906$
|  |  | S | $62.470\,000\,000\,000\,006$ | $-0.388\,369$ | $-2.675\,921$
|  |  | Y | $49.13$ | $0.043\,114$ | $-2.094\,973$
| GSM8k | $49.355\,572$ | B | $99.38$ | $-15.269\,369$ | $-24.754\,392$
|  |  | S | $7.199\,999\,999\,999\,999$ | $4.060\,143$ | $-0.588\,852$
|  |  | Y | $1.94$ | $4.518\,178$ | $1.044\,871$
| Hellaswag | $80.900\,000$ | B | $99.26$ | $-2.660\,342$ | $-4.353\,652$
|  |  | S | $44.87$ | $0.151\,874$ | $-1.701\,369$
|  |  | Y | $94.78$ | $-1.483\,063$ | $-2.942\,781$
| MMLU | $61.162\,228$ | S | $4.18$ | $3.011\,799$ | $0.177\,771$
|  |  | Y | $10.879\,999\,999\,999\,999$ | $1.781\,875$ | $-0.627\,897$
Llama-3-Instruct-70b | ARC | $70.221\,843$ | B | $28.64$ | $1.451\,209$ | $-2.804\,033$
|  |  | S | $75.48$ | $-1.371\,385$ | $-5.121\,566$
|  |  | Y | $68.86$ | $-0.516\,492$ | $-2.794\,930$
| GSM8k | $89.992\,418$ | B | $73.619\,999\,999\,999\,99$ | $-1.207\,465$ | $-7.535\,637$
|  |  | S | $9.39$ | $3.054\,342$ | $-1.025\,125$
|  |  | Y | $20.0$ | $1.225\,975$ | $-1.501\,201$
| Hellaswag | $85.300\,000$ | B | $18.970\,000\,000\,000\,002$ | $0.872\,838$ | $-0.881\,624$
|  |  | S | $97.729\,999\,999\,999\,99$ | $-2.449\,629$ | $-4.375\,215$
|  |  | Y | $89.67$ | $-1.239\,536$ | $-2.672\,832$
| MMLU | $77.530\,266$ | S | $43.269\,999\,999\,999\,996$ | $0.173\,751$ | $-3.562\,850$
|  |  | Y | $15.82$ | $1.179\,238$ | $-1.068\,549$
Llama-3-Instruct-8b | ARC | $60.750\,853$ | B | $45.04$ | $0.248\,494$ | $-3.767\,514$
|  |  | S | $66.869\,999\,999\,999\,99$ | $-0.766\,205$ | $-3.882\,902$
|  |  | Y | $44.330\,000\,000\,000\,005$ | $0.198\,431$ | $-2.087\,818$
| GSM8k | $75.890\,826$ | B | $7.93$ | $8.404\,302$ | $-1.204\,805$
|  |  | S | $23.07$ | $1.492\,501$ | $-2.245\,498$
|  |  | Y | $70.850\,000\,000\,000\,01$ | $-0.890\,681$ | $-3.630\,570$
| Hellaswag | $78.100\,000$ | B | $99.460\,000\,000\,000\,01$ | $-3.021\,217$ | $-4.945\,848$
|  |  | S | $98.960\,000\,000\,000\,01$ | $-2.959\,448$ | $-4.852\,172$
|  |  | Y | $90.89$ | $-1.431\,390$ | $-3.177\,164$
| MMLU | $62.227\,603$ | S | $76.69$ | $-1.168\,432$ | $-3.795\,250$
|  |  | Y | $27.51$ | $0.741\,684$ | $-1.528\,227$
Table 16: Contamination results for Phi-2, Phi-3-Mini, Phi-3-Small, Phi-3-Medium. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Phi-2 | ARC | $58.447\,099$ | B | $70.14$ | $-1.287\,620$ | $-5.432\,313$
|  |  | S | $97.63$ | $-4.513\,072$ | $-7.795\,974$
|  |  | Y | $53.03$ | $-0.078\,348$ | $-2.392\,314$
| GSM8k | $58.908\,264$ | B | $<10^{-2}$ | $36.418\,186$ | $26.464\,750$
|  |  | S | $56.18$ | $-0.259\,532$ | $-4.025\,092$
|  |  | Y | $28.199\,999\,999\,999\,996$ | $1.039\,166$ | $-2.094\,119$
| Hellaswag | $76.300\,000$ | B | $4.64$ | $3.462\,670$ | $0.080\,791$
|  |  | S | $100.0$ | $-4.797\,412$ | $-6.727\,351$
|  |  | Y | $26.31$ | $0.829\,163$ | $-1.682\,386$
| MMLU | $51.961\,259$ | S | $99.39$ | $-5.009\,148$ | $-7.866\,099$
|  |  | Y | $40.5$ | $0.323\,253$ | $-2.161\,636$
Phi-3-Mini | ARC | $59.897\,611$ | B | $59.699\,999\,999\,999\,996$ | $-0.612\,314$ | $-4.717\,489$
|  |  | S | $99.78$ | $-6.943\,849$ | $-10.597\,187$
|  |  | Y | $49.33$ | $0.050\,037$ | $-2.505\,304$
| GSM8k | $76.648\,976$ | B | $0.29$ | $16.298\,102$ | $6.328\,980$
|  |  | S | $89.75$ | $-2.860\,105$ | $-6.288\,625$
|  |  | Y | $86.350\,000\,000\,000\,01$ | $-1.734\,688$ | $-4.288\,113$
| Hellaswag | $80.550\,000$ | B | $1.15$ | $3.380\,323$ | $1.174\,736$
|  |  | S | $100.0$ | $-6.341\,258$ | $-8.373\,729$
|  |  | Y | $62.19$ | $-0.345\,902$ | $-2.259\,739$
| MMLU | $63.486\,683$ | S | $99.009\,999\,999\,999\,99$ | $-4.939\,916$ | $-8.047\,090$
|  |  | Y | $40.400\,000\,000\,000\,006$ | $0.291\,757$ | $-2.150\,364$
Phi-3-Small | ARC | $67.832\,765$ | B | $74.839\,999\,999\,999\,99$ | $-1.878\,597$ | $-6.620\,246$
|  |  | S | $63.09$ | $-0.756\,393$ | $-4.548\,810$
|  |  | Y | $26.75$ | $1.154\,486$ | $-1.843\,096$
| GSM8k | $87.945\,413$ | B | $0.58$ | $15.040\,878$ | $5.040\,354$
|  |  | S | $61.3$ | $-0.698\,768$ | $-5.039\,457$
|  |  | Y | $58.489\,999\,999\,999\,995$ | $-0.364\,773$ | $-3.310\,678$
| Hellaswag | $84.700\,000$ | B | $0.02$ | $5.405\,288$ | $3.383\,966$
|  |  | S | $81.51$ | $-1.141\,480$ | $-3.266\,680$
|  |  | Y | $71.57$ | $-0.599\,506$ | $-2.401\,998$
| MMLU | $71.331\,719$ | S | $89.8$ | $-2.888\,308$ | $-6.670\,263$
|  |  | Y | $40.6$ | $0.374\,828$ | $-2.291\,204$
Phi-3-Medium | ARC | $66.552\,901$ | B | $83.99$ | $-2.728\,280$ | $-7.157\,697$
|  |  | S | $87.56$ | $-2.683\,533$ | $-6.566\,132$
|  |  | Y | $91.94$ | $-2.565\,694$ | $-5.545\,608$
| GSM8k | $86.353\,298$ | B | $0.03$ | $21.113\,273$ | $10.775\,327$
|  |  | S | $37.34$ | $0.763\,470$ | $-3.230\,430$
|  |  | Y | $65.149\,999\,999\,999\,99$ | $-0.661\,989$ | $-3.590\,123$
| Hellaswag | $86.050\,000$ | B | $0.02$ | $4.541\,280$ | $2.509\,587$
|  |  | S | $92.44$ | $-1.831\,291$ | $-3.874\,381$
|  |  | Y | $20.51$ | $0.871\,330$ | $-0.894\,879$
| MMLU | $74.140\,436$ | S | $63.190\,000\,000\,000\,005$ | $-0.725\,399$ | $-4.637\,474$
|  |  | Y | $51.349\,999\,999\,999\,994$ | $-0.038\,856$ | $-2.638\,105$
Table 17: Contamination results for Mistral-7b-v0.1, Mistral-7b-v0.2. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Mistral-7b-v0.1 | ARC | $58.959\,044$ | B | $95.009\,999\,999\,999\,99$ | $-4.455\,115$ | $-8.763\,969$
|  |  | S | $7.91$ | $2.210\,294$ | $-0.395\,809$
|  |  | Y | $28.07$ | $0.792\,550$ | $-1.521\,839$
| GSM8k | $39.044\,731$ | B | $87.59$ | $-7.707\,288$ | $-18.036\,196$
|  |  | S | $0.15$ | $8.245\,073$ | $4.479\,598$
|  |  | Y | $66.44$ | $-0.867\,632$ | $-4.424\,772$
| Hellaswag | $83.650\,000$ | B | $65.34$ | $-0.420\,786$ | $-2.192\,001$
|  |  | S | $0.24$ | $3.135\,526$ | $1.271\,238$
|  |  | Y | $16.97$ | $0.942\,815$ | $-0.682\,913$
| MMLU | $58.014\,528$ | S | $15.229\,999\,999\,999\,999$ | $1.880\,507$ | $-1.051\,506$
|  |  | Y | $11.08$ | $1.877\,129$ | $-0.686\,812$
Mistral-7b-v0.2 | ARC | $58.191\,126$ | B | $98.02$ | $-5.626\,025$ | $-10.007\,729$
|  |  | S | $23.24$ | $1.165\,086$ | $-1.379\,672$
|  |  | Y | $45.61$ | $0.153\,305$ | $-2.114\,602$
| GSM8k | $37.604\,246$ | B | $89.880\,000\,000\,000\,01$ | $-8.549\,284$ | $-18.872\,100$
|  |  | S | $0.54$ | $6.570\,893$ | $2.677\,350$
|  |  | Y | $76.32$ | $-1.531\,027$ | $-5.069\,130$
| Hellaswag | $82.800\,000$ | B | $58.14$ | $-0.246\,970$ | $-2.110\,882$
|  |  | S | $1.28$ | $2.598\,258$ | $0.687\,073$
|  |  | Y | $50.080\,000\,000\,000\,005$ | $-0.011\,898$ | $-1.600\,226$
| MMLU | $56.707\,022$ | S | $49.62$ | $0.062\,126$ | $-2.866\,919$
|  |  | Y | $27.939\,999\,999\,999\,998$ | $0.888\,364$ | $-1.605\,434$
Table 18: Contamination results for Mistral-Instruct-7b-v0.3, Mistral-Instruct-7b-v0.2, Mistral-Instruct-7b-v0.1. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Mistral-Instruct-7b-v0.1 | ARC | $54.351\,536$ | B | $99.97$ | $-9.039\,077$ | $-12.923\,015$
|  |  | S | $95.1$ | $-2.825\,126$ | $-5.398\,847$
|  |  | Y | $73.29$ | $-0.863\,918$ | $-3.170\,674$
| GSM8k | $35.860\,500$ | B | $98.509\,999\,999\,999\,99$ | $-13.888\,893$ | $-23.858\,595$
|  |  | S | $85.289\,999\,999\,999\,99$ | $-2.449\,291$ | $-5.630\,621$
|  |  | Y | $90.91$ | $-2.673\,429$ | $-5.827\,728$
| Hellaswag | $74.900\,000$ | B | $90.11$ | $-1.680\,136$ | $-3.966\,215$
|  |  | S | $97.119\,999\,999\,999\,99$ | $-2.471\,928$ | $-4.623\,176$
|  |  | Y | $53.04$ | $-0.134\,858$ | $-2.514\,433$
| MMLU | $49.685\,230$ | S | $29.29$ | $0.773\,960$ | $-1.764\,685$
|  |  | Y | $72.16$ | $-0.874\,259$ | $-3.359\,791$
Mistral-Instruct-7b-v0.2 | ARC | $62.457\,338$ | B | $0.04$ | $10.619\,953$ | $5.953\,197$
|  |  | S | $64.89$ | $-0.701\,681$ | $-4.171\,914$
|  |  | Y | $80.34$ | $-1.286\,835$ | $-3.800\,503$
| GSM8k | $42.608\,036$ | B | $80.76$ | $-5.710\,978$ | $-16.014\,867$
|  |  | S | $94.39$ | $-4.261\,145$ | $-8.354\,931$
|  |  | Y | $84.91$ | $-2.100\,872$ | $-5.240\,427$
| Hellaswag | $84.550\,000$ | B | $0.18$ | $3.524\,014$ | $1.562\,859$
|  |  | S | $81.26$ | $-1.041\,316$ | $-3.009\,596$
|  |  | Y | $12.47$ | $1.104\,502$ | $-0.560\,605$
| MMLU | $55.060\,533$ | S | $18.63$ | $1.478\,764$ | $-1.276\,184$
|  |  | Y | $42.67$ | $0.249\,200$ | $-2.212\,040$
Mistral-Instruct-7b-v0.3 | ARC | $62.372\,014$ | B | $88.82$ | $-3.242\,017$ | $-7.591\,791$
|  |  | S | $94.57$ | $-3.636\,688$ | $-7.378\,530$
|  |  | Y | $66.69$ | $-0.616\,358$ | $-3.056\,531$
| GSM8k | $48.218\,347$ | B | $94.33$ | $-10.604\,234$ | $-20.555\,309$
|  |  | S | $35.14$ | $1.030\,959$ | $-3.811\,940$
|  |  | Y | $89.74$ | $-2.692\,001$ | $-6.148\,853$
| Hellaswag | $84.250\,000$ | B | $5.2$ | $1.895\,925$ | $-0.017\,023$
|  |  | S | $62.580\,000\,000\,000\,005$ | $-0.397\,431$ | $-2.424\,326$
|  |  | Y | $37.49$ | $0.279\,341$ | $-1.307\,692$
| MMLU | $56.561\,743$ | S | $14.149\,999\,999\,999\,999$ | $1.878\,497$ | $-0.934\,840$
|  |  | Y | $34.01$ | $0.628\,017$ | $-1.886\,374$
Table 19: Contamination results for Mixtral-Instruct-8x22b, Mixtral-Instruct-8x7b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
Mixtral-Instruct-8x22b | ARC | $72.952\,218$ | B | $0.77$ | $6.008\,639$ | $2.001\,147$
|  |  | S | $10.63$ | $2.637\,773$ | $-1.074\,578$
|  |  | Y | $20.07$ | $1.102\,772$ | $-1.423\,721$
| GSM8k | $85.974\,223$ | B | $44.6$ | $0.693\,421$ | $-3.858\,890$
|  |  | S | $65.36$ | $-0.680\,065$ | $-4.370\,882$
|  |  | Y | $21.060\,000\,000\,000\,002$ | $1.184\,303$ | $-1.459\,922$
| Hellaswag | $89.000\,000$ | B | $14.01$ | $1.192\,130$ | $-0.620\,198$
|  |  | S | $22.49$ | $0.817\,587$ | $-1.127\,950$
|  |  | Y | $26.36$ | $0.617\,782$ | $-1.251\,953$
| MMLU | $74.140\,436$ | S | $29.849\,999\,999\,999\,998$ | $0.972\,688$ | $-2.574\,199$
|  |  | Y | $59.95$ | $-0.234\,595$ | $-2.416\,769$
Mixtral-Instruct-8x7b | ARC | $69.539\,249$ | B | $73.09$ | $-1.493\,789$ | $-5.589\,857$
|  |  | S | $47.27$ | $0.108\,493$ | $-3.476\,509$
|  |  | Y | $68.300\,000\,000\,000\,01$ | $-0.547\,881$ | $-2.851\,769$
| GSM8k | $65.655\,800$ | B | $95.56$ | $-8.844\,977$ | $-18.061\,643$
|  |  | S | $69.87$ | $-1.055\,162$ | $-4.530\,944$
|  |  | Y | $83.789\,999\,999\,999\,99$ | $-1.781\,700$ | $-4.629\,638$
| Hellaswag | $87.600\,000$ | B | $20.79$ | $0.821\,752$ | $-0.849\,242$
|  |  | S | $14.57$ | $1.145\,737$ | $-0.763\,073$
|  |  | Y | $6.310\,000\,000\,000\,000\,5$ | $1.341\,872$ | $-0.098\,606$
| MMLU | $66.731\,235$ | S | $26.75$ | $1.066\,656$ | $-1.934\,444$
|  |  | Y | $77.86$ | $-0.975\,355$ | $-3.092\,340$
Table 20: Contamination results for StableLM-2-12b, StableLM-2-Zephyr-3b, StableLM-2-1.6b, StableLM-2-Alpha-7b-v2. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
StableLM-2-Alpha-7b-v2 | ARC | $44.283\,276$ | B | $100.0$ | $-10.654\,478$ | $-15.059\,631$
|  |  | S | $60.46$ | $-0.583\,613$ | $-3.993\,027$
|  |  | Y | $45.14$ | $0.136\,415$ | $-2.762\,688$
| GSM8k | $5.307\,051$ | B | $61.89$ | $-1.100\,844$ | $-7.508\,739$
|  |  | S | $27.98$ | $0.648\,926$ | $-1.120\,145$
|  |  | Y | $90.92$ | $-1.715\,644$ | $-3.803\,337$
| Hellaswag | $77.200\,000$ | B | $100.0$ | $-5.409\,374$ | $-7.361\,438$
|  |  | S | $3.88$ | $2.651\,460$ | $0.229\,271$
|  |  | Y | $43.34$ | $0.147\,842$ | $-2.177\,329$
| MMLU | $42.130\,751$ | S | $3.2$ | $3.998\,800$ | $0.492\,295$
|  |  | Y | $26.450\,000\,000\,000\,003$ | $1.113\,448$ | $-1.865\,036$
StableLM-2-Zephyr-3b | ARC | $44.795\,222$ | B | $23.74$ | $5.198\,786$ | $-10.165\,451$
|  |  | S | $75.44$ | $-1.425\,371$ | $-4.930\,972$
|  |  | Y | $56.379\,999\,999\,999\,995$ | $-0.319\,878$ | $-3.545\,475$
| GSM8k | $51.630\,023$ | B | $<10^{-2}$ | $48.780\,356$ | $44.338\,575$
|  |  | S | $3.29$ | $5.467\,228$ | $0.541\,929$
|  |  | Y | $75.39$ | $-1.448\,760$ | $-4.920\,258$
| Hellaswag | $72.950\,000$ | B | $35.74$ | $0.743\,137$ | $-3.177\,716$
|  |  | S | $100.0$ | $-7.454\,460$ | $-9.833\,778$
|  |  | Y | $44.96$ | $0.146\,735$ | $-3.667\,903$
| MMLU | $40.290\,557$ | S | $83.87$ | $-1.609\,592$ | $-4.447\,917$
|  |  | Y | $35.870\,000\,000\,000\,005$ | $0.633\,019$ | $-2.458\,091$
StableLM-2-1.6b | ARC | $43.430\,034$ | B | $100.0$ | $-26.354\,077$ | $-31.679\,197$
|  |  | S | $14.899\,999\,999\,999\,999$ | $2.411\,374$ | $-2.407\,163$
|  |  | Y | $56.100\,000\,000\,000\,01$ | $-0.299\,912$ | $-3.487\,304$
| GSM8k | $18.877\,938$ | B | $<10^{-2}$ | $16.560\,354$ | $12.953\,915$
|  |  | S | $23.94$ | $1.217\,834$ | $-1.638\,261$
|  |  | Y | $49.96$ | $0.001\,648$ | $-2.915\,218$
| Hellaswag | $71.050\,000$ | B | $100.0$ | $-5.159\,122$ | $-7.605\,729$
|  |  | S | $5.19$ | $4.312\,090$ | $-0.057\,506$
|  |  | Y | $31.06$ | $1.017\,350$ | $-2.926\,092$
| MMLU | $35.690\,073$ | S | $29.73$ | $1.247\,481$ | $-2.335\,381$
|  |  | Y | $32.269\,999\,999\,999\,996$ | $0.843\,138$ | $-2.200\,500$
StableLM-2-12b | ARC | $59.470\,990$ | B | $99.97$ | $-14.013\,736$ | $-19.284\,169$
|  |  | S | $0.679\,999\,999\,999\,999\,9$ | $4.607\,515$ | $1.593\,007$
|  |  | Y | $1.099\,999\,999\,999\,999\,9$ | $3.783\,061$ | $1.116\,257$
| GSM8k | $57.998\,484$ | B | $0.389\,999\,999\,999\,999\,96$ | $17.921\,418$ | $6.961\,541$
|  |  | S | $11.92$ | $3.104\,906$ | $-1.337\,474$
|  |  | Y | $17.98$ | $1.845\,436$ | $-1.507\,611$
| Hellaswag | $84.450\,000$ | B | $18.16$ | $0.975\,840$ | $-0.869\,374$
|  |  | S | $2.96$ | $2.157\,709$ | $0.247\,046$
|  |  | Y | $30.69$ | $0.486\,172$ | $-1.149\,802$
| MMLU | $56.513\,317$ | S | $77.34$ | $-1.461\,150$ | $-4.671\,489$
|  |  | Y | $72.72$ | $-0.977\,053$ | $-3.704\,123$
Table 21: Contamination results for StableLM-2-Instruct-12b, StableLM-2-Instruct-1.6b. B is benchmark-specific, S is sample-specific and Y is syntax-specific contamination. All numbers are reported in percentages. Model | Benchmark | Perf. [%] | Type |  $p$ [%] |  $\hat{\delta}$ [%] |  $\hat{\delta}_{0.95}$ [%]
---|---|---|---|---|---|---
StableLM-2-Instruct-1.6b | ARC | $42.662\,116$ | B | $29.38$ | $5.534\,772$ | $-19.026\,879$
|  |  | S | $70.33$ | $-1.202\,027$ | $-5.112\,418$
|  |  | Y | $48.97$ | $-0.002\,118$ | $-3.385\,619$
| GSM8k | $42.001\,516$ | B | $<10^{-2}$ | $27.791\,320$ | $19.268\,710$
|  |  | S | $26.27$ | $1.767\,553$ | $-2.668\,701$
|  |  | Y | $62.760\,000\,000\,000\,005$ | $-0.673\,934$ | $-4.318\,263$
| Hellaswag | $69.600\,000$ | B | $31.25$ | $1.084\,094$ | $-4.377\,379$
|  |  | S | $100.0$ | $-5.989\,269$ | $-8.501\,862$
|  |  | Y | $68.679\,999\,999\,999\,99$ | $-1.207\,934$ | $-5.207\,004$
| MMLU | $38.305\,085$ | S | $61.21$ | $-0.660\,640$ | $-4.046\,503$
|  |  | Y | $9.370\,000\,000\,000\,001$ | $2.503\,834$ | $-0.635\,551$
StableLM-2-Instruct-12b | ARC | $64.249\,147$ | B | $2.17$ | $26.109\,032$ | $5.201\,972$
|  |  | S | $67.88$ | $-1.145\,295$ | $-5.280\,809$
|  |  | Y | $18.72$ | $1.610\,878$ | $-1.311\,145$
| GSM8k | $68.840\,030$ | B | $<10^{-2}$ | $32.932\,918$ | $21.094\,482$
|  |  | S | $17.740\,000\,000\,000\,002$ | $2.371\,387$ | $-1.863\,203$
|  |  | Y | $32.26$ | $0.883\,581$ | $-2.409\,537$
| Hellaswag | $86.250\,000$ | B | $<10^{-2}$ | $7.037\,107$ | $4.882\,316$
|  |  | S | $98.4$ | $-3.428\,354$ | $-5.843\,002$
|  |  | Y | $3.32$ | $2.181\,370$ | $0.240\,805$
| MMLU | $55.496\,368$ | S | $90.13$ | $-2.494\,296$ | $-5.672\,264$
|  |  | Y | $44.42$ | $0.246\,261$ | $-2.741\,016$