Strong Model Collapse
Under review as a conference paper at ICLR 2025

STRONG MODEL COLLAPSE

Anonymous authors
Paper under double-blind review

ABSTRACT

Within the scaling laws paradigm, which underpins the training of large neural
networks like ChatGPT and Llama, we consider a supervised regression setting
and establish a strong form of the model collapse phenomenon, a critical perfor-
mance degradation due to synthetic data in the training corpus. Our results show
that even the smallest fraction of synthetic data (e.g., as little as 1 per 1000) can
still lead to model collapse: larger and larger training sets do not enhance perfor-
mance. We further investigate whether increasing model size, an approach aligned
with current trends in training large language models, exacerbates or mitigates
model collapse. In a simplified regime where neural networks are approximated
via random projections of tunable size, we both theoretically and empirically show
that larger models can amplify model collapse. Interestingly, our theory also in-
dicates that, beyond the interpolation threshold (which can be extremely high for
very large datasets), larger models may mitigate the collapse, although they do
not entirely prevent it. Our theoretical findings are empirically verified through
experiments on language models and neural networks for images.

1

INTRODUCTION

The term Model Collapse refers to a critical degradation in the performance of AI models, particu-
larly when a significant portion of their training data consists of synthetic data generated by other
models. As detailed in Shumailov et al. (2023), this phenomenon arises as the model gradually over-
fits to patterns found in synthetic data, which may not fully represent the richness or variability of
real-world data. Over successive training cycles, this feedback loop results in the model reinforcing
errors, biases, or oversimplifications from the synthetic data. Consequently, the model’s ability to
generalize to real-world data is compromised, as it increasingly relies on the distorted distribution
provided by prior AI generations rather than learning accurate representations of the real world.

This phenomenon was observed empirically (Hataya et al., 2023; Mart´ınez et al., 2023a;b; Bohacek
& Farid, 2023; Briesch et al., 2023; Guo et al., 2023) and described theoretically (Alemohammad
et al., 2023; Bertrand et al., 2023; Dohmatob et al., 2024a;b). The connection to the breakdown
of neural scaling laws (Kaplan et al., 2020) has been pointed out and analyzed in Dohmatob et al.
(2024b): as data becomes more synthetic, larger training sets do not enhance performance.

The issue is especially concerning in large-scale AI systems like ChatGPT and Llama (Touvron
et al., 2023; Dubey & et al., 2024), which rely heavily on vast amounts of training data to maintain
their performance. If synthetic data is used in training these models, even in small quantities, the
model can start producing “gibberish” or nonsensical outputs, contains misinformation, or reflect
stereotypes. This is because the model effectively starts to amplify its own mistakes (Shumailov
et al., 2024). This feedback loop results in a gradual loss of model fidelity, reducing its ability to
generalize or adapt to new, unseen test environments.

1.1 MAIN CONTRIBUTIONS

In this work, we establish a series of results which shed more light on model collapse, bringing the
phenomenon closer to a solid theoretical foundation. We consider the following important questions:

(Q1) Is model collapse inevitable or can it be fixed by strategically mixing synthetic and real data?
(Q2) Are larger models more prone to model collapse than smaller ones?

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

Our theoretical analysis focuses on the solvable setting of linear regression with and without random
projections, with the latter serving as an approximation of neural networks by means of random
feature maps (Maloney et al., 2022; Bach, 2023). Also, in accordance with the current “neural
scaling laws” paradigm (Kaplan et al., 2020; Hoffmann et al., 2022) whichs underlies the training
of LLMs, where models and dataset sizes become larger over time, we focus on the setup where the
total amount of data (synthetic + real data) used for training grows arbitrarily.

Let us summarize our main findings.
– Result #1: Strong Model Collapse. First, we establish a robust negative result which shows that
model collapse generally persists even when mixing real and synthetic data, as long as the fraction
of training data which is synthetic does not vanish (cf. Section 3.1 and 3.2). By synthetic data, we
mean any training data from a distribution which deviates from the distribution of real data, i.e. data
on which the test performance is evaluated. Thus, model collapse cannot generally be mitigated by
simple adjustments such as data weighting (Jain et al., 2024; Ferbach et al., 2024) unless these strate-
gies asymptotically remove all but a vanishing proportion of synthetic data from the training process
(Section 5). Our results show that the findings of Shumailov et al. (2024); Alemohammad et al.
(2023); Bertrand et al. (2023); Dohmatob et al. (2024a;b) are worse than anticipated, by considering
the more realistic scenario where a mixture of synthetic and real data is used for training.

Figure 1: Pareto diagram: Understanding the role of model size in model collapse. We compare the test
error (on the real / true data distribution), for a random projections model (equation (5) of Section 2.2) when
training is done on a mix of synthetic and real data (y-axis), versus real data only (x-axis); in both cases, the
total amount of training data is fixed to n = 500. On the scatter plots, square points correspond to very high-
quality synthetic data (i.e from a distribution which is close to the true data distribution), diamonds correspond
to high-quality synthetic data, triangles correspond to low-quality, while stars correspond to very low-quality
synthetic data. The black lines correspond to the Pareto frontiers for each level of quality of the synthetic data;
the higher the frontier above the diagonal in the given setting, the more serious is the model collapse. The
colorbar is the log of parametrization rate ψ = m/n, where m captures is the size of the model.

– Result #2: Model Size and Model Collapse. In Section 3.2, we disentangle the effect of a model’s
size on its ability to cope with model collapse. We show that in general, bigger models will suffer
more from model collapse as soon as the deviation between the distribution of the synthetic data
and real data is significant. Crucially, our theory also predicts that past the interpolation threshold
point, this tendency can be reversed:
large models become more robust to model collapse. Put
together, these results predict the existence of a double-descent curve regarding the model collapse
phenomenon. This is illustrated in Figures 1 and 2. Thus, the model collapse profile depends
critically on design choices like model size.

Experimental Validation. Our theoretical results are empirically confirmed with experiments in :

• Toy settings, including random projections model on Gaussian data, and shallow networks fully
trained on the MNIST dataset (Deng, 2012). Refer to the end of Section 3.2 and Appendix A.2.
• Realistic setting of GPT-2 models trained on BabiStories (Zhang et al., 2024a), a reproduction
of TinyStories (Eldan & Li, 2023) using the Mixtral-8x7B open language model (Jiang et al.,
2024)). Refer to Section 4.

Approach. From a technical standpoint, our theoretical analysis focuses on regression problems
in the classical linear setting introduced in Dohmatob et al. (2024a) for studying model collapse,
and also the setting of neural networks in a simplified regime which can be approximated by ran-
dom projections (Maloney et al., 2022; Bach, 2023). We employ the tools of operator-valued free
probability theory (OVFPT) (Mingo & Speicher, 2017) to obtain a new bias-variance decomposition

2

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

0.20.40.6Test error (train on real data only)323130Test error (train on real + synth.)synthetic dataset size n2=500.20.40.6Test error (train on real data only)synthetic dataset size n2=2000.20.40.6Test error (train on real data only)synthetic dataset size n2=4000.500.250.000.250.50loge()Under review as a conference paper at ICLR 2025

Figure 2: Illustration of our new bias-variance decomposition Etest ≃ B + V + ζ for neural networks
in the simplified random projections regime (cf. Section 3.2), trained on a mixture of real and synthetic data.
The sum B + V corresponds to the classical bias variance decomposition in this setup when all the training
data is real. The extra term ζ is responsible for model collapse when training is done on a mixture of real and
synthetic data. The scalar c2 characterizes the quality of the synthetic data (cf. Definition 1), via its mismatch
with the real data distribution. The vertical line corresponds to the interpolation threshold m = n, where m is
the model size and n is the total sample size. Notice the well-known double-descent curve in the bias curve.

Etest ≃ B + V + ζ, of the test error evaluated on the real / true data distribution, of a model trained
on a mixture of real and synthetic data. The extra term ζ then induces model collapse.

1.2 RELATED WORK

The theoretical study of model collapse in the setting of high-dimensional supervised-learning with
linear regression and kernel ridge regression was initiated in Dohmatob et al. (2024a). This work
derives analytical formulas that quantitatively describe iterative retraining on synthetic data in both
under-parameterized and over-parameterized regimes, considering both low- and high-dimensional
asymptotics. It places itself within an important body of works studying kernel ridge regression
(on “clean” data), which serves as an effective proxy for neural networks in various regimes, for
instance in the infinite-width limit (Neal, 1996; Williams, 1996; Jacot et al., 2018; Lee et al., 2018)
or in the lazy regime of training (Chizat et al., 2019) and are a testbed to study interesting phenomena
observed in deep learning. For instance, Rahimi & Recht (2008); Rudi & Rosasco (2017); Maloney
et al. (2022) study scaling laws for regression in the random feature model and Bach (2023) analyses
double descent in this setting. Scaling laws have been shown for kernel models under the Gaussian
design, e.g. in Caponnetto & de Vito (2007); Spigler et al. (2020); Cui et al. (2022) for regression
and Cui et al. (2023) for classification.

Very few theoretical works tackle the analysis of models trained on mixtures of original (real /
clean) and synthetic data. Bertrand et al. (2023) analyze the training process at the distribution level
and provide stability results under a locality assumption in parameter space. Seddik et al. (2024)
analyze the mixing of discrete original and synthetic data, and provide upper bounds on the amount
of synthetic data that can be included to avoid model collapse. Let us also mention the recent works
(Jain et al., 2024; Ferbach et al., 2024) which are potential methods for mitigating model collapse.
Jain et al. (2024) analyze linear regression on isotropic Gaussian data for mixtures of clean and
synthetic data by minizing a strategically weighted sum of losses (one term for each data source,
real and synthetic), while Ferbach et al. (2024) can be seen as a multi-step version thereof where at
each stage, the synthetic data generator is distilled by interpolating with real data. These methods
are analyzed in Section 5, where we outline their shortcomings regarding model collapse.

Finally, a few works go beyond the mixing scenario and analyze how to curate or filter synthetic
data to avoid model collapse (Feng et al., 2024; Zhang et al., 2024b; Alemohammad et al., 2024),
but a rigorous study of their effectiveness is still lacking.

2 THEORETICAL SETUP

2.1 DATA DISTRIBUTIONS

Consider an iid sample from D1 = {(xi, yi) | 1 ≤ i ≤ n1} of size n1 from the true data distribution
P1 and an independent iid sample D2 = {(xi, yi) | n1 + 1 ≤ i ≤ n} of size n2 from another data
distribution P2 (which we shall hereafter call the synthetic data distribution), where n := n1 + n2
is the total amount of training data. Here, Pk = PΣk,w∗
(Features) x ∼ N (0, Σk),
(Labels) y = x⊤w∗

is the distribution on Rd × R given by

k + ϵ, with ϵ ∼ N (0, σ2

k) independent of x.

k,σ2
k

(1)

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

02505007501000Network width m0.00.20.50.81.0Bias Bc20.00.10.51.002505007501000Network width m0.00.20.50.81.0Variance V02505007501000Network width m0.02.04.06.0Extra term 02505007501000Network width m0.02.04.06.0Test error EtestUnder review as a conference paper at ICLR 2025

Each Σk is a d × d positive-definite covariance matrix which captures the intrinsic variations of the
input feature vector x. The σk’s control the level of label noise in each distribution.
Structure of the Label Shift. For conciseness, we will assume the following priors on the w∗

k’s

• True labelling function: w∗
• Mismatch between real and synthetic: δ := w∗

1 ∼ N (0, Γ),

2 − w∗

1 ∼ N (0, ∆), independent of w∗
1,

for some d × d positive-semidefinite matrices Γ and ∆.

Remark 1. To ease the presentation of our results, we shall assume that the matrices Σ1, Σ2, Γ, and
∆ are diagonal matrices, and therefore commute. Furthermore, except otherwise explicitly stated,
we shall assume equal covariance matrices, and take Σ1 = Σ2 = Σ as in Dohmatob et al. (2024a).

1 and σ2

The matrix Γ captures the structure of the ground-truth labelling function in the real / test distribution
P1. Together with the label-noise levels σ2
1) captures the
covariance structure of the disparity between the true data distribution P1 and the synthetic data
distribution P2 regarding the conditional distribution p(y|x); the marginal distribution of x stays the
same under P1 and P2 due the assumption Σ1 = Σ2 = Σ. For example, the self-consuming-loops
setup of Dohmatob et al. (2024a) corresponds to taking ∆ proportional to the precision matrix of
the input features Σ−1. Thus, the size of the fluctuations of each component δj of the difference
w∗
1 is inversely proportional to the standard deviation of the corresponding feature. Another
important setup is the case where the fluctuations are isotropic, i.e taking ∆ ∝ Id.

2, the matrix ∆ = cov(w∗

2 − w∗

2 − w∗

Quality of Synthetic Data. Due to the a priori general structure of ∆, the label corresponding to an
input x will be different for both distributions, even in the absence of label-noise. On average, the
L2-norm of this difference is Ew∗
2)2] = tr Σ∆. We therefore define
Definition 1. The quality of synthetic data is defined as c2(∆) = (1/d) tr Σ∆, which captures the
disparity between the synthetic data distribution P2 and the real data distribution P1 (small values
of c2(∆) are better). For example, if ∆ = c2Σ−1 as in Dohmatob et al. (2024a), then c2(∆) = c2.

Ex∼N (0,Σ) [(x⊤w∗

1 − x⊤w∗

1 ,w∗
2

2.2 MODELS AND PERFORMANCE MEASURE

Given this training data, the goal of a learner is to construct an estimator (cid:98)w. This can be seen as a
linear model from x (cid:55)→ x⊤
(cid:98)w. Evaluated on the real / true data distribution P1 (which coincides with
the distribution from which the real component D1 of the training dataset D is drawn), the test error
of a model (cid:98)f : Rd → R is defined by

Etest( (cid:98)f ) = EDEx∼N (0,Σ1)[( (cid:98)f (x) − x⊤w∗

1)2].

(2)

This will be our main object of study, for different models (cid:98)f . The outermost expectation ED is to
quench the randomness in the training dataset D used to train the model.

We consider two families of analytically tractable models: (1) classical linear models obtained via
penalized regression in the input space, and (2) models obtained via penalized regression in a feature
space given by random projections. The latter allows us to study the role of model size in model
collapse, by varying the output dimension of the random projection mapping. This output dimension
m controls the size of a neural network in a simplified regime (Maloney et al., 2022; Bach, 2023).

(1) Classical Linear Model. We start with a setup motivated by Dohmatob et al. (2024a). We are
interested in the penalized linear model (ridge) (cid:98)fCL : x (cid:55)→ x⊤
(cid:98)w with parameter vector (cid:98)w given by

(cid:98)w = arg min
w∈Rd

1
n

n
(cid:88)

(x⊤

i w − yi)2 + λ∥w∥2,

i=1

(3)

trained on the total dataset D = D1 ∪ D2. Of course, the unregularized limit λ → 0+ corresponds to
ordinary least-squares (OLS). We shall work in the following so-called proportionate scaling limit

(Proportionate Scaling Limit for Classical Linear Model) For fixed ϕ ∈ (0, ∞), p2 ∈ (0, 1),

d, n, n1, n2 → ∞, n2/n → p2, n1/n → p1 = 1 − p2,

d/n → ϕ.

(4)

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

The extreme cases p1 → 0+ and p2 → 0+ correspond to training on only synthetic (resp. real) data.
In particular, p1 → 0+ corresponds to the setting considered in Dohmatob et al. (2024a). Note that
in the isotropic setting where Σ ∝ Id, while ϕ controls the speed of learning on clean data. Indeed,
for small ϕ, the scaling law in this case is known (Hastie et al., 2022) to be Etest ≃ σ2
1ϕ + O(ϕ2).
As we shall see (Corollary 1), this scaling law gets deformed in the presence of synthetic data in the
training dataset, leading to model collapse.

(2) Random Projections Model. We consider neural networks in a simplified regime which can be
approximated via random projections (Maloney et al., 2022; Bach, 2023), i.e f (x) = x⊤Sv. Here,
S is a d × m random matrix with iid entries from N (0, 1/d); it maps an input-vector x ∈ Rd to a
random feature vector z = Φ(x) := S⊤x ∈ Rm. Only the “read-out” weights v ∈ Rm are learned,
by fitting on the dataset D. Consider the model (cid:98)fRP : x (cid:55)→ Φ(x)⊤

(cid:98)v, where (cid:98)v is given by

(cid:98)v = arg min
v∈Rm

1
n

n
(cid:88)

(v⊤Φ(xi) − yi)2 + λ∥v∥2.

i=1

(5)

Note that such a simplified neural network model has been proposed in the literature as a theoretical
testbed for studying intriguing properties of neural networks, like scaling laws (Maloney et al.,
2022) and double-descent (Bach, 2023). Also see Section 1.2. It can be shown that the extreme case
m/n → ∞ reduces to the classical linear model.

We shall work in the following asymptotic regime:

(Proportionate Scaling Limit for Random Projections Model)

d, m, n, n1, n2 → ∞, n1/n → p1, n2/n → p2,

d/n → ϕ, m/d → γ, m/n → ψ,

(6)

for some constants ϕ, γ, ψ ∈ (0, ∞) and p1, p2 ∈ (0, 1), with p1 + p2 = 1 and ψ = ϕγ.

Note that the ratio ψ/ϕ ≃ md captures the size of the network, though the number of trainable
parameters (the read-out layer) is m ≃ γd.

3 A NEW BIAS-VARIANCE DECOMPOSITION AND
THE EMERGENCE OF STRONG MODEL COLLAPSE

3.1 CLASSICAL LINEAR MODELS

We begin with an analysis of the test error Etest( (cid:98)fCL) for the classical linear model defined in
(3) trained on a mixture of synthetic and true / real data, but evaluated on test data from the true
data distribution only. We will establish a new bias-variance decomposition with an additional term
which quantitatively reveals the emergence of model collapse (Shumailov et al., 2023; 2024).

Let us first recall some standard notations. Denote by κ = κ(n, λ; Σ) the unique positive solution
to the fixed-point equation

κ − λ = κ df 1(κ; Σ)/n, with df k(t; Σ) := tr Σk(Σ + tId)−k.

(7)

Also define u = u(n, λ; Σ) ≥ 0 as follows u :=

df 2(κ; Σ)/n
1 − df 2(κ; Σ)/n

.

The following result (proved in the appendix, alongside all other theoretical results in this work)
will be exploited in the sequel to show that the use of synthetic data in model training can lead to
catastrophic effects regarding test error.
Theorem 1. Define σ2 := p1σ2
2 and let κ, u ≥ 0 be as previously constructed. In the
proportionate scaling limit (4), the test error w.r.t the true data distribution P1, of the classical
linear model (cid:98)fCL defined in (3) is given by Etest( (cid:98)fCL) ≃ E + ζ, with

1 + p2σ2

E = B + V, V = σ2 df 2(κ; Σ)/n

1 − df 2(κ; Σ)/n

, B = κ2 tr ΓΣ(Σ + κId)−2
1 − df 2(κ; Σ)/n

,

ζ = p2

2 · (1 + p1u) tr ∆Σ3(Σ + κId)−2 + p2u tr ∆Σ(p1Σ + κId)2(Σ + κId)−2.

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

2 = w∗

Note that for ∆ = 0 (i.e w∗
1), which corresponds to assuming that the real data and the
surrogate data have the same distribution, the above theorem gives Etest( (cid:98)fCL) ≃ E ≃ B +V which
is the classical bias-variance decomposition (Hastie et al., 2022; Richards et al., 2021) for ridge
1 ,σ2. The extra term ζ appearing in Theorem 1
regression on n samples from the distribution PΣ,w∗
is responsible for model collapse! In Appendix D.2, we show how Theorem 1 recovers the main
results of Dohmatob et al. (2024a) for special choices of the displacement matrix ∆.
Strong Model Collapse. In particular, in the “scaling laws” regime where ϕ → 0+, it holds that
ζ ≃ p2
2 tr ∆. In this case, if tr ∆ remains bounded away from zero, then so is ζ unless p2 → 0+,
i.e we discard all synthetic data from the training dataset. This is strong model collapse. It hints
that model collapse as exposed by Shumailov et al. (2023; 2024); Hataya et al. (2023); Mart´ınez
et al. (2023a;b); Bohacek & Farid (2023); Briesch et al. (2023); Guo et al. (2023) cannot be fixed by
naively mixing synthetic and real data during training. We show in Section 3.2 that this observation
continues to hold in the setting of random projections model (cid:98)fRP defined in (5). Finally, in Section 5
we study what happens when the synthetic data and real data are strategically mixed during training.

1 , X2, or X ⊤

Proving Theorem 1.
It turns out that the analysis of the classical linear model’s test error
Etest( (cid:98)fCL) in Theorem 1 amounts to the analysis of the trace of rational functions of sums of
random matrices. Although the limiting spectral density of sums of random matrices is a classi-
cal computation using subordination techniques (Marˇcenko & Pastur, 1967; Kargin, 2015), this is
not enough for the full analysis; a more involved analysis is required. For example, some of the
quantities we must analyze are of the following form (where Mj := X ⊤
j Xj/n, M := M1 + M2;
A and B deterministic matrices): r(3)
j (A, B) := E tr AMj(M + λId)−1B(M + λId)−1Mj. The
difficulty will be even greater in the setting of random projections (cid:98)fRP because it leads to more
complicated terms. To the rescue, in Appendix E we shall employ operator-valued free probabil-
ity theory (OVFPT) to compute the exact high-dimensional limits of quantities like the definition
of r(3)
j (A, B) above . The tools of OVFPT have been used in the recent machine learning theory
literature to obtain precise asymptotics for the test error of neural networks (trained on real data
only) in various linearized settings (Adlam & Pennington, 2020; Tripuraneni et al., 2021; Lee et al.,
2023). The idea is to construct a block matrix Q each of whose blocks is a constant or is propor-
tional X1, X ⊤
2 , and one of the blocks Q−1[i, j] of Q−1 is equal to the original matrix
W = AMj(M + λId)−1B(M + λId)−1Mj. Such a Q is referred to as a linear pencil for W .
Because of the structure of Q, OVPT allows us to compute the limiting value of the expectation as
the traces of the square blocks of Q−1, and we ultimately extract r(3)
j (A, B) ≃ lim E tr Q−1[i, j].
Example: The Isotropic Case. To help unpack Theorem 1, consider the following concrete setup
Σ = Id, Γ = (r2/d)Id, ∆ = (c2/d)Id,
for some constants r, c > 0. The constant c2 captures how close the distribution of the synthetic
data P2 is to the distribution of the real data P1; thus it captures the quality of the synthetic data.
This gives u ≃ ϕ/((1 + κ)2 − ϕ), where κ > 0 uniquely satisfies the fixed-point equation κ − λ =
κϕ/(1 + κ); this is a quadratic equation that characterizes the well-known Marchenko-Pastur law
(Marˇcenko & Pastur, 1967). The quantities appearing in the formulae presented in Theorem 1 then
take the following simple forms: V = σ2ϕ/((1 + κ)2 − ϕ) and B = κ2r2/((1 + κ)2 − ϕ), and
ζ = (cid:0)p2(1 + p1u) + (p1 + κ)2u(cid:1) p2c2/(1 + κ)2.
In particular, in the unregularized limit λ → 0+ corresponding to OLS, we get κ → (ϕ − 1)+.
To further make things concrete, consider the under-parametrized case where ϕ ∈ (0, 1) in the
proportionate scaling regime (4). The over-parametrized case ϕ ∈ (1, ∞) is treated in Appendix
D.1. We deduce the following corollary to Theorem 1.
Corollary 1. Suppose ϕ ∈ (0, 1). Then, in the limit (4) and λ → 0+, the test error with respect to
the true data distribution P1, of the classical linear model (cid:98)fCL defined in (3) is given by

Etest( (cid:98)fCL) ≃ σ2ϕ/(1 − ϕ) + (cid:0)p2

2 + p2p1ϕ/(1 − ϕ)(cid:1) c2.

Moreover, for fixed c > 0 and small ϕ ∈ (0, 1), it holds that Etest( (cid:98)fCL) ≃ σ2d/n + p2
2c2 + O(ϕ2).
In particular, if c2 = Ω(1), i.e bounded away from zero (corresponding to low-quality synthetic
data), then Etest( (cid:98)fCL) = Ω(p2
2c2): the scaling law plateaus unless p2 → 0+, i.e unless all but a
vanishing proportion of synthetic data is discarded from the training dataset.

6

Under review as a conference paper at ICLR 2025

Figure 3: Strong model collapse in classical linear model (empirical confirmation of Corollary 1). The
training dataset comprises of n = n1 + n2 samples from a mixture of n2 = p2n synthetic samples and n1 =
n − n2 real samples. The real samples are from the same distribution as the real / true samples of the training
dataset, while the synthetic samples are from a distribution with the same covariance structure and label noise
level σ = 1, but an incorrect labelling function (epistemic error). The quality of the synthetic data is controlled
by the scalar c, with c → 0 corresponding to synthetic data of perfect quality (higher values correspond to
lower quality synthetic data). Solid curves correspond to experiments, and broken curves correspond to our
theoretical predictions of Corollary 1; notice the perfect match. We see that even a small amount of low-quality
synthetic data is enough to cause model collapse, whereby the test error of the model deviates from a perfect
diagonal (ideal scaling law, corresponding to p2 = 0, i.e training on real data only).

The result is empirically illustrated in Figure 3, where we see that even a small amount of low-quality
synthetic data is enough to cause model collapse, whereby the test error of the model deviates from
a perfect diagonal (ideal scaling law, corresponding to p2 = 0, i.e training on real data only).
Remark 2. Corollary 1 can be extended to the the non-isotropic case, but the statement is much
longer and is thus omitted here.

3.2 RANDOM PROJECTIONS MODEL

We now turn to the more challenging setting of the random projections model (cid:98)fRP given in (5). As
mentioned before (cf. Section 2.2), such models are considered in our work as a simplification of
the high-dimensional dynamics of actual neural networks, which still allows us to capture the effect
of model size in the model collapse phenomenon.

We will need the following scalars which capture the high-dimensional statistics of the model (cid:98)fRP .
Definition 2. Let (e, τ, u, ω) be the unique positive solution to the following fixed-point equations

1/e = 1 + ψτ ¯tr ΣK −1,

1/τ = 1 + ¯tr K0K −1, with K0 := eΣ, K := γτ K0 + λId,

u = ψe2 ¯tr Σ(γτ 2L′ + ωId)K −2, ω = τ 2 ¯tr (γωK 2

0 + λ2L′)K −2, with L′ := (1 + u)Σ.

(8)

(9)

Here, ¯tr A := tr A/d is the normalized trace. Also define θ := λ/(γτ e) > 0, ω′ := ω/(γτ 2) > 0.

1 + p2σ2

As usual, we denote σ2 := p1σ2
2. Also, for any p ∈ [0, 1] define a d × d positive definite
matrix T (θ; p) := pΣ+θId, and T (θ) := T (θ; p)|p=1 = Σ+θId. The following result is a nontrivial
extension of Theorem 1 to the case of random projections.
Theorem 2. In the proportionate scaling limit (6), the test error w.r.t the true data distribution P1,
of the random projections model (cid:98)fRP defined in (5) is given by Etest( (cid:98)fRP ) ≃ E + ζ, with
E ≃ B + V, where B = (1 + u)θ2 tr ΓΣT (θ)−2 + ω′ tr ΓΣ2T (θ)−2,
V = (cid:0)tr Σ2T (θ)−2 + (ω′ − θu) tr ΣT (θ)−2(cid:1) σ2/e ,
ζ = p2(p2 + p1u) tr ∆Σ3T (θ)−2 + p2(ω′ + 2p1uθ) tr ∆Σ2T (θ)−2 + p2uθ2 tr ∆ΣT (θ)−2.

(10)

We now explore a few important consequences of Theorem 2.

A Double-Descent Curve. The bias-variance decomposition presented in Theorem 2 is empirically
illustrated in Figures 2 and 4 for the Gaussian setting (1) (see Appendix B.1 and A.1 for details
on the experimental setup and additional results in this setting). Notice the perfect match with
experiments. The shape of the bias curve in Figure 2 (leftmost plot) is reminiscent of the well-
known double-descent (Bach, 2023) in the unregularized setting λ → 0+. The divergence at the
interpolation threshold m = n (i.e. ψ = 1) is because the bias term B, the variance term V , and the
extra term ζ (responsible for model collapse) all diverge to infinity at this point.

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Strong Model Collapse. Observe that the first term in the expression for ζ given in Theorem 2 is
lower-bounded by p2
2 tr ∆Σ3(Σ + θId)−2, which scales linearly with the square of the proportion
p2 ≃ n2/n of synthetic data in the training dataset D. However, unless p2 → 0+, i.e unless
the proportion p2 of synthetic data in the training dataset vanishes, the performance of the model
eventually plateaus above the baseline E (corresponding to the setting where all training data is real,
i.e no synthetic data in training dataset). This is strong model collapse.
Since the factor tr ∆Σ3(Σ + θId)−2 only depends on the
design choices of the model (via the scalar θ defined pre-
viously), we expect that different design choices (e.g.,
model size) will lead to different model collapse profiles.

Are Larger Models More Prone or Less Prone to
Model Collapse? Figure 1 shows the results of a small
experiment to investigate this. The input dimension
is d = 600, and the covariance matrix is identity Id
(isotropic features). The total number of training exam-
ples is fixed to n = 500. The ∆ matrix is taken to be of
the form ∆ = (c2/d)Σ−1 (similar results are observed for
different covariance matrices) for different values of c2
as follows: c2 = 0 (synthetic data of very high quality),
represented with square markers; c2 = 0.1 (high qual-
ity synthetic data), represented with diamonds; c2 = 0.5
(low quality), represented by triangles; and c2 = 1 (very
low-quality synthetic data), represented by stars. As indi-
cated on the figure, the leftmost plot corresponds to the regime where there is much fewer synthetic
than real samples (n2 = 50 synthetic samples versus n1 = 450 real samples). Here, for both very
high-quality and high-quality (squares and diamonds), the optimal tradeoff is struck by larger models
(i.e, larger values of ψ). For lower-quality data (triangles and stars), the frontier shifts upwards and
from left to right; intermediately sized models become optimal for coping with model collapse.

Figure 4: Impact of model size (network
width m) on model collapse. As usual,
solid curves correspond to experimental re-
sults (5 runs), while broken curves corre-
spond to predictions of our theory (here,
Corollary 4). Error bars correspond to 5 in-
dependent runs. Also see Figures 2 and 7.

In the middle plot, size of the synthetic dataset is comparable to the size of the real dataset (n2 = 200
versus n1 = 300). For high-quality synthetic data, larger models are still better than smaller models.
However, for this setting, the frontier shifts upwards and from left to right, and the optimal model
size is intermediate. For the rightmost plot, the size of the synthetic dataset is considerably larger
than the real dataset (n2 = 400 versus n1 = 100). The results are similar to the case n2 = 200
except that the Pareto frontiers are higher over the diagonal (indicating more serious model collapse).
In all cases, very small models are never optimal: they are not good even in the classical sense when
training is done only on real data, and the presence of synthetic data only makes this worse.
Special Cases of Theorem 2 In the limit p2 → 0+ (i.e., no synthetic data; all the training data is
real), ζ → 0 in Theorem 2, and we recover the main result of Bach (2023) as a special case, namely
Etest( (cid:98)fRP ) ≃ B + V , with B and V as given in the theorem. Note that even in this special case, our
result is more general since it covers the entire regularization path while the formulae in Bach (2023)
are only for the unregularized case λ → 0+. On the other hand, Theorem 2 is a generalization of
Theorem 1, as can be seen by taking ψ → ∞. Refer to Appendix G.2 for details.

4 EXPERIMENTAL RESULTS

Our theoretical framework is developed within the context of high-dimensional linear regression and
random projections models using Gaussian data. Our first departure from the confines of our theory
are experiments with two-layer neural networks trained on the MNIST dataset (Deng, 2012) both
in the random feature model (with ReLU activations) and with fully trained networks. These are
presented in Appendix A.2. We find that the general trends observed in our asymptotic theory still
hold: (1) there is significant model collapse, which only diminishes as the fraction of synthetic data
approaches 0; (2) larger models exhibit a more severe model collapse (Figures 8 and 9).

We now provide evidence that our theory is applicable to large-scale problems, particularly in the
context of language modeling with GPT-2 models. The BabiStories dataset (Zhang et al., 2024a), a
reproduction of TinyStories (Eldan & Li, 2023) using the Mixtral-8x7B open language model (Jiang

8

02505007501000Network width m0123456Test error Etestc20.00.10.51.0Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Figure 5: Results on BabiStories with GPT-2 models. Synthetic BabiStories is generated with a trained GPT-
2-small with the same set of prompts. (Left) Impact of the proportion of synthetic data p2 on model collapse
in a language model with 12 layers. (Right) Impact of model size (number of layers) on model collapse. Here
the model is trained on synthetic data only (i.e p2 = 1). The loss is evaluated on the TinyStories test set.

et al., 2024) enables us to study language modeling with relatively small models in a compute-
efficient and environmentally friendly way. It comprises stories generated by prompting large mod-
els to create narratives in simple language that a three-year-old child could understand, effectively
simplifying the underlying language model.

Setup. We train a GPT-2-small model with 124 million parameters on the BabiStories dataset as the
generator. Using the same story prompts, which include the story beginning and character names, the
generator creates our synthetic dataset. We then mix this synthetic dataset with the original BabiS-
tories, train, and evaluate model perplexity on a validation set derived from the original BabiStories.
Detailed information on optimization and model configurations is provided in Appendix B.3.

Impact of Synthetic Data Proportion. We investigate the effect of varying the synthetic data
proportion (p2) on the model’s scaling in Figure 5 (left). Here, the x-axis represents the number of
tokens used to train the model. In this experiment, the synthetic data is of high quality, as evidenced
by the low training loss and coherent text generations, corresponding to the small c2 (cf. Definition
1) case in our illustrative Figure 1. Consequently, even moderate amounts of synthetic data delay
the progression of the scaling laws, and we expect this to eventually lead to plateaus or at least very
bad bad (i.e small) exponents in the final scaling laws as predicted in Dohmatob et al. (2024b) in the
special case of training on synthetic data only.

Impact of Model Size. We next examine the impact of model size on training with synthetic data. In
addition to the GPT-2-small model (12 layers), we introduce two larger models: one with 18 layers
(166 million parameters) and another with 24 layers (204 million parameters). The embedding
dimension and number of attention heads remain constant across all models. We generate a synthetic
dataset 10 times larger than the original one to support the scaling of tokens. As shown in Figure
5 (right), larger (deeper) models maintain a lower test loss until the dataset size increases—likely
exceeding the interpolation threshold—at which point smaller models begin to exhibit lower loss
and reduced overfitting. This aligns with the predictions of Theorem 2 (also refer to Figure 1, 2,
and the discussion just after the theorem), which suggest that larger models tend to amplify model
collapse beyond the interpolation threshold. In Figure 5, we observe this amplification when the
number of tokens exceeds 3 × 1010. Conversely, the theory predicts that over-parameterized models
help mitigate collapse, a trend we observe when the number of tokens is below 1 × 1010, leading to
improved performance of larger models.

5 CAN STRATEGIC DATA MIXING SCHEMES PREVENT MODEL COLLAPSE?

Having established the occurrence of strong model collapse both theoretically and empirically, we
now explore strategies to mitigate it and leverage synthetic data under stronger assumptions. We
begin by assuming clear information about the data source and consider the following strategic
iterative mixing, inspired by Ferbach et al. (2024). In this approach, a model is fitted on a mixture
of synthetic and real data. In the next iteration, the labels for the synthetic data are replaced with the
labels predicted by the previous iteration of the process, and so on.
For concreteness, take Σ1 = Σ2 = Σ = Id for the covariance matrices, and ∆ = (c2/d)Σ−1 =
(c2/d)Id. In this setup, the proposal of Ferbach et al. (2024) then becomes

• The quality parameter (cf. Def 1) of the synthetic data at the beginning (i.e at t = 0) is c2 = c2
0.

9

1096×1082×1093×1094×109Number of Tokens1.5×1001.6×1001.7×1001.8×1001.9×1002×100Lossp20.00.0010.0050.010.020.050.10.20.50.91.010910101011Number of Tokens1.65×1001.7×1001.75×1001.8×1001.85×1001.9×1001.95×1002×100LossNumber of Layers121824Under review as a conference paper at ICLR 2025

Figure 6: Iterative vs Single-Step Mixing. Solid lines represent the experimental results (5 runs), while
dashed lines correspond to the theoretical predictions of Corollary 2. The iterative mixing is repeated 5 times,
with p1 = p2 = 0.5. “Clean” refers to the scaling when using solely the n1 = p1n real data in the dataset.

• At iteration t + 1, we mix n2 = p2n samples of synthetic data from a source having quality
t , with n1 = n − n2 samples of real data to construct a penalized linear model

parameter c2 = c2
(cid:98)w(t+1) according to (3). This trained model generates the synthetic data with c2 = c2
t+1.
Thus, the idea is to iteratively enhance the quality of the synthetic data through bootstrapping.
Corollary 2. For large t, it holds in the limit (4) and then ϕ, λ → 0+ that

Etest( (cid:98)w(t)) ≃ E/(1 − p2

2) + Θ(p2t

2 ), where E = σ2ϕ/(1 − ϕ) ≃ σ2d/n.

(11)

We now explore some important consequences of Corollary 2.

Strategic Mixing Recovers Scaling Laws but Might Not be Feasible in Practice. If the practi-
tioner can curate a sufficient amount of data from the original distribution, the training dataset will
include a non-vanishing proportion of real data, ensuring that p1 remains bounded away from zero.
By comparing E with p2t
2 , we observe that iterative mixing over t iterations, where t is of the order
of log(n/d), results in a scaling law proportional to E, as empirically confirmed in Figure 6. How-
ever, this comes at the cost of significant bootstrapping, a large volume of real data, and the need
to clearly distinguish between real and synthetic data across iterations—conditions that are all too
computationally expensive and challenging to implement in practice.

Iterative Mixing with Little Real Data is Bad. If we consider the setting where we only have
limited real data or where there is faster accumulation of synthetic data, which corresponds to p2 →
1 (the real data in the training set is diminishing), then it holds that for any t ≥ 1, Etest( (cid:98)w(t)) ≃
c2
0 + tE. This is an increasing function of t, meaning that there is still catastrophic model collapse.
Single-Step Mixing Does Not Avoid Model Collapse. We demonstrate in Appendix C that single-
step mixing of real and synthetic data (i.e t = 1 iteration) exhibits strong model collapse, even when
the mixing coefficient for each component of the dataset is optimally tuned as proposed in (Jain
et al., 2024), an approach which can be seen as an instance of weighted empirical risk minimization
(ERM) Shimodaira (2000); Vogel et al. (2021).

6 DISCUSSION

Our work systematically characterizes the effects of training models on mixtures of real and syn-
thetic data, showing that model collapse is a robust phenomenon that persists even with small frac-
tions of synthetic data, in the asymptotic regime. By introducing new mathematical tools, we extend
prior work to analyze more complex mixing settings and models (random projections), broadening
the scope of theoretically tractable problems. Experiments confirm our theoretical predictions across
large language models (LLMs) and also fully-trained feed-forward neural networks.

Going beyond the prevalent “neural scaling laws” paradigm (Kaplan et al., 2020; Hoffmann et al.,
2022) which is at the basis of the current trend in training LLMs, this study emphasizes the impor-
tance of preserving and labeling real data, either by curating it or avoiding unintended synthetic data
in training, reflecting a shift as AI-generated data becomes prevalent. Our work also delineates the
impact of model size on the model collapse profile. Future work will explore the effect of other
model design choices like activation functions, depth, and optimization hyper-parameters like learn-
ing rate and momentum. To this end, we can leverage “Gaussian equivalents” (Goldt et al., 2022) to
extend our theory to wide, fully-trained networks in the neural tangent kernel (Jacot et al., 2018) and
lazy (Chizat et al., 2019) regimes, using operator-valued free probability theory (Mingo & Speicher,
2017), like we have done in our analysis.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

103104Total dataset size n102101100Test errorc20=100MixingIterativeSingleClean103104Total dataset size nc20=101103104Total dataset size nc20=102Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent
and a multi-scale theory of generalization. In International Conference on Machine Learning, pp.
74–84. PMLR, 2020.

Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein
Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative
models go mad. arXiv preprint arxiv:2307.01850, 2023.

Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John Collomosse, and Richard
Baraniuk. Self-improving diffusion models with synthetic data, 2024. URL https://arxiv.
org/abs/2408.16333.

Francis Bach. High-dimensional analysis of double descent for linear regression with random pro-

jections. 2023.

Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier
Gidel. On the stability of iterative retraining of generative models on their own data. arXiv
preprint arxiv:2310.00429, 2023.

Matyas Bohacek and Hany Farid. Nepotistically trained generative-ai models collapse, 2023.

Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their

own output: An analysis of the self-consuming training loop, 2023.

Andrea Caponnetto and Ernesto de Vito. Optimal rates for the regularized least-squares algorithm.

Foundations of Computational Mathematics, 7:331–368, 2007.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming.

Advances in neural information processing systems, 32, 2019.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborov´a. Generalization error rates
the crossover from the noiseless to noisy regime. Journal of Statistical

in kernel regression:
Mechanics: Theory and Experiment, 2022(11):114004, nov 2022.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborov´a. Error scaling laws for kernel
classification under source and capacity conditions. Machine Learning: Science and Technology,
4(3):035033, August 2023. ISSN 2632-2153. doi: 10.1088/2632-2153/acf041.

Li Deng. The mnist database of handwritten digit images for machine learning research [best of the

web]. IEEE signal processing magazine, 29(6):141–142, 2012.

Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regres-

sion. arXiv preprint arXiv:2402.07712, 2024a.

Elvis Dohmatob, Yunzhen Feng, Pu Yang, Franc¸ois Charton, and Julia Kempe. A tale of tails:
Model collapse as a change of scaling laws. In Forty-first International Conference on Machine
Learning, 2024b. URL https://openreview.net/forum?id=KVvku47shW.

Abhimanyu Dubey and et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak

coherent english? arXiv preprint arXiv:2305.07759, 2023.

Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model col-
lapse: Scaling up with synthesized data requires reinforcement, 2024. URL https://arxiv.
org/abs/2406.07515.

Damien Ferbach, Quentin Bertrand, Avishek Joey Bose, and Gauthier Gidel. Self-Consuming Gen-

erative Models with Curated Data Provably Optimize Human Preferences. ArXiv, 2024.

Damien Ferbach, Quentin Bertrand, Avishek Joey Bose, and Gauthier Gidel. Self-consuming gen-
erative models with curated data provably optimize human preferences, 2024. URL https:
//arxiv.org/abs/2407.09499.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc M´ezard, and Lenka Zde-
borov´a. The gaussian equivalence of generative models for learning with shallow neural networks.
In Mathematical and Scientific Machine Learning, pp. 426–471. PMLR, 2022.

Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chlo´e Clavel. The curious decline of

linguistic diversity: Training language models on synthetic text, 2023.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high-

dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2), 2022.

Ryuichiro Hataya, Han Bao, and Hiromi Arai. Will large-scale generative models corrupt future
datasets? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
pp. 20555–20565, October 2023.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen-
nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy,
Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.
Training compute-optimal large language models, 2022.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and gen-
eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-
Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31.
Curran Associates, Inc., 2018.

Ayush Jain, Andrea Montanari, and Eren Sasoglu. Scaling laws for learning with real and surrogate

data, 2024. URL https://arxiv.org/abs/2402.04376.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam-
ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child,
Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language
models. arXiv preprint arXiv:2001.08361, 2020.

V. Kargin. Subordination for the sum of two random matrices. The Annals of Probability, 43(4):

2119 – 2150, 2015.

Donghwan Lee, Behrad Moniri, Xinmeng Huang, Edgar Dobriban, and Hamed Hassani. Demys-
In International Conference on Machine

tifying disagreement-on-the-line in high dimensions.
Learning, pp. 19053–19093. PMLR, 2023.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha
Sohl-Dickstein. Deep neural networks as gaussian processes. In 6th International Conference on
Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Confer-
ence Track Proceedings. OpenReview.net, 2018.

Alexander Maloney, Daniel A. Roberts, and James Sully. A solvable model of neural scaling laws,

2022.

Gonzalo Mart´ınez, Lauren Watson, Pedro Reviriego, Jos´e Alberto Hern´andez, Marc Juarez, and
Rik Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards
evolution or degradation? arXiv preprint arxiv: 2303.01255, 2023a.

Gonzalo Mart´ınez, Lauren Watson, Pedro Reviriego, Jos´e Alberto Hern´andez, Marc Juarez, and Rik
Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet.
arXiv preprint arxiv: 2306.06130, 2023b.

V.A. Marˇcenko and Leonid Pastur. Distribution of eigenvalues for some sets of random matrices.

Math USSR Sb, 1:457–483, 01 1967.

James A. Mingo and Roland Speicher. Free Probability and Random Matrices, volume 35 of Fields

Institute Monographs. Springer, 2017.

12

Under review as a conference paper at ICLR 2025

Radford M. Neal. Priors for infinite networks.

In Bayesian Learning for Neural Networks, pp.

29–53. Springer, New York, 1996.

Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization
with randomization in learning. In Advances in Neural Information Processing Systems. Curran
Associates, Inc., 2008.

Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge(less) regression
under general source condition. In Proceedings of The 24th International Conference on Artificial
Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research. PMLR,
2021.

Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features.
ISBN

In Advances in Neural Information Processing Systems. Curran Associates Inc., 2017.
9781510860964.

Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Deb-
bah. How bad is training on synthetic data? a statistical analysis of language model collapse.
arXiv preprint arXiv:2404.05090, 2024.

H. Shimodaira. Improving predictive inference under covariate shift by weighting the loglikelihood

function. Journal of Statistical Planning and Inference, 90(2):227–244, 2000.

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander-
son. The curse of recursion: Training on generated data makes models forget. arXiv preprint
arxiv:2305.17493, 2023.

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson.

Ai models collapse when trained on recursively generated data. Nature, 631, 2024.

Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods:
empirical data versus teacher–student paradigm. Journal of Statistical Mechanics: Theory and
Experiment, 2020(12):124001, December 2020.
ISSN 1742-5468. doi: 10.1088/1742-5468/
abc61d.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko-
lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda-
tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Covariate shift in high-dimensional random

feature regression. arXiv preprint arXiv:2111.08234, 2021.

Robin Vogel, Mastane Achab, St´ephan Cl´emenc¸on, and Charles Tillier. Weighted empirical risk
minimization: Sample selection bias correction based on importance sampling. ArXiv, 2021.

Christopher Williams. Computing with infinite networks. In M.C. Mozer, M. Jordan, and T. Petsche

(eds.), Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996.

Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, and L´eon Bottou. Memory mosaics.

arXiv preprint arXiv:2405.06394, 2024a.

Jinghui Zhang, Dandan Qiao, Mochen Yang, and Qiang Wei. Regurgitative training: The value of

real data in training large language models. arXiv preprint arXiv:2407.12835, 2024b.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

Appendix

Table of Contents

A Further Experimental Results

A.1 Additional Results for the toy setting of multivariate gaussians
.
A.2 Experimental results for Neural Networks on MNIST .
.
.
A.3 General Picture for Neural Network on MNIST .

.
.

.
.

.
.

.

.

B Experimental Details

B.1 Toy Setting: Random Projections Model
.
B.2 Two-layer neural networks
.
.
B.3 Language Modeling .

.
.

.
.

.
.

.
.

.
.

.

.

.
.
.

.
.
.

.
.
.

.
.
.

.
.
.

.
.
.

.
.
.

.
.
.

.
.
.

.
.
.

.
.
.

.
.

.
.

.
.
.

C Static / Single-Step Data Mixing

D Some Omitted Theoretical Results and Comments

D.1 Classical Linear Model in Over-Parametrized Isotropic Setting .
.
D.2 Connections to Classical Model Collapse in Regression .

.

.

E Deterministic Equivalents

.
.
.

E.1 Classical Linear Model
.
E.2 Random Projections .
E.3 Proof of Proposition 1 .
Computing r(1)
. .
Computing r(4). .
Computing r(3). .
Computing r(2). .
.

E.4 Proof of Proposition 2 .

j

.
.
.

.

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.

.

.
.
.
.

F Proof of Theorem 1 and Corollaries
.
.
.
.
.

F.1 Proof of Theorem 1 .
F.2 Proof of Corollary 1 .
F.3 Proof of Corollary 3 .
F.4 Proof of Corollary 2 .
F.5 Proof of Corollary 4 .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

G Proof of Proposition 2 and Theorem 2
.

G.1 Proof of Proposition 2 .
.
G.2 Recovering Theorem 1 from Theorem 2 .
.
G.3 Proof of Theorem 2 .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

H Phase-Diagram for Random Projections Model
.
.

H.1 The General Regularized Case .
.
H.2 Unregularized Limit .

.
.

.
.

.
.

.
.

.
.

.

.

.

.

I Raw Bias-Variance Decomposition
.
.

I.1 Classical Linear Model
.
I.2 Random Projections Model

.

.
.

.
.

.
.

.
.

.
.

.
.

.
.

14

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

.
.
.

.
.
.

.
.

.
.
.

.

.
.
.
.

.
.
.
.
.

.
.
.

.
.

.
.

15
15
15
16

17
17
17
17

18

19
19
19

20
20
21
21

21

22
24
24
24

25
25
26
26
27
27

28
28
30
30

30
30
31

32
32
33

Under review as a conference paper at ICLR 2025

A FURTHER EXPERIMENTAL RESULTS

A.1 ADDITIONAL RESULTS FOR THE TOY SETTING OF MULTIVARIATE GAUSSIANS

Figure 7 provides additional plots for various data quality parameters c2 showing model collapse
as a function of model size in the toy setting of multivariate Gaussians with random projections
(experimental details in Section B.1).

Figure 7: Impact of model size (network width m) on model collapse. Same setting as for Figure 4, but
with quality parameter c2 (smaller is better) as shown on top of each plot and proportion of synthetic data p2
as in the legend (Figure 4 showed the reverse).

A.2 EXPERIMENTAL RESULTS FOR NEURAL NETWORKS ON MNIST

Setup. For two-layer neural networks, we consider
two scenarios: (1) learning with a random projection
model as in Section 3.2, where the first layer of the
network is fixed randomly, and only the second layer
is trained, and (2) learning with a fully trainable neu-
ral network. The first setting directly corresponds to
our theoretical results from Section 3.2, but with ReLU
activation functions. In the case of fully trained neu-
ral networks in the second setting, our theory does not
apply directly. However, we hypothesize that the gen-
eral trends observed in our asymptotic theory will still
hold: (1) there will be a significant model collapse,
which only diminishes as the fraction of synthetic data
approaches 0; (2) larger models will exhibit a more se-
vere model collapse.

Figure 8: Fully trained two-layer network
on MNIST data.
Impact of model size (hid-
den dimension, aka network width) on model
collapse. Here, the model is trained solely on
synthetic data (i.e p2 → 1).

To align with the theoretical setting, we employ a (multivariate) regression approach where labels
are converted to one-hot vectors and the model is trained using mean squared error. The synthetic
labels were generated by another two-layer network, with Gaussian label noise (standard deviation
of 0.1) added. A validation set is used to select the best checkpoint, and evaluation is conducted on
the test set using the clean labels. Further details of the training are provided in Appendix B.2.

Results. Figure 9 presents the results for both random feature models (left) and fully trained neural
networks (right). In these experiments, we mixed synthetic and original data in the training set with
varying coefficients, p1. As the proportion of synthetic data, p2, increases, the scaling laws slow
down and eventually plateau. We observe a strong model collapse: only when p2 approaches 0
does the collapse subside. The results are consistent across both cases, validating our theoretical
predictions and demonstrating the applicability of our insights to more complex scenarios.

We also investigated how model size, specifically the hidden dimension of fully trained neural net-
works, affects model collapse. As shown in Figure 8, models with varying hidden dimensions were
trained exclusively on the synthetic dataset with p2 = 1. For training sets ranging from 10,000 to
50,000 samples, our results indicate that larger models are more susceptible to model collapse under
the same validation and evaluation protocols. Notably, all these models remain in the interpolation
regime, aligning with our theoretical predictions.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

02004006008001000Network width m0.00.20.40.60.81.0Test errorc2=0.0p20.020.10.20.40.60.802004006008001000Network width m0.00.20.40.60.81.0c2=0.102004006008001000Network width m0.00.51.01.52.02.53.0c2=0.502004006008001000Network width m0.00.51.01.52.02.53.0c2=1.0103104Number of Samples1016×102MSE LossHidden Dimension10040010002000400010000Under review as a conference paper at ICLR 2025

Figure 9: Model collapse as a function of the proportion of synthetic data. We use the MNIST dataset with
regression loss. Error bars correspond to 5 runs. Left, Random feature model with hidden dimension 100,000.
Right, Two-layer neural network of width (i.e hidden dim.) m = 2000.

Figure 10: Understanding the role of model size in model collapse under varying qualities of
synthetic data and dataset sizes. The quality of the synthetic data is evaluated using the MSE loss
on the test set. The model is trained solely on synthetic data (p2 → 1).

A.3 GENERAL PICTURE FOR NEURAL NETWORK ON MNIST

To provide a comprehensive understanding of how the quality of synthetic data, the quantity of
synthetic data, and the network size impact performance, we conducted a large-scale experiment
varying these factors, as shown in Figures 10 and 11. Figure 10 uses the MSE loss as an indicator of
synthetic data quality, while Figure 11 uses accuracy as the indicator. To simplify the analysis, we
focus on pure synthetic data (p2 = 1).

The synthetic data with the highest quality already achieves accuracy close to the optimal. As the
quality decreases, we observe that the shape of the curve begins to resemble a double descent curve,
similar to the changes in the Pareto frontiers shown in Figure 1. With different combinations of
the number of synthetic data n2 and hidden dimension d, the figure captures various segments of
the double descent curve depicted in Figure 4. When n2 is small (as seen in the left subplots), it
corresponds to a large parameterization rate ψ, placing it in the second descent region of the double
descent curve. Conversely, when n2 is large (as shown in the right subplots), it captures the up-and-
down behavior characteristic of the double descent phenomenon.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

102103104Number of Samples102101MSE Loss102103104Number of SamplesThe proportion of synthetic data, p200.0010.0010.0050.010.020.050.10.20.50.9102103104Hidden Dimension0.050.100.15MSE LossSynthetic size n2 = 2000102103104Hidden DimensionSynthetic size n2 = 10000102103104Hidden DimensionSynthetic size n2 = 40000Quality of Synthetic Data (Generator Performance)0.0210.0570.0950.148Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Figure 11: Understanding the role of model size in model collapse under varying qualities of
synthetic data and dataset sizes. The quality of the synthetic data is evaluated using the accuracy
on the test set. The model is trained solely on synthetic data (p2 → 1).

B EXPERIMENTAL DETAILS

B.1 TOY SETTING: RANDOM PROJECTIONS MODEL

Setup. As a sanity check to empirical confirm our analytical predictions from Theorem 2, we con-
sider a setting with multivariate Gaussian data (1). The feature covariance matrix Σ is constructed
to have power-law eigenvalues λj = C/j, where C is such that tr Σ = λ1 + . . . + λd = 1. The
ground-truth labelling weights w∗
1 of the real data distribution P1 sampled from N (0, (1/d)Id),
while the ground-truth weights w∗
2 for the synthtic data distribution are sampled from N (w∗
1, ∆)
with ∆ = (c2/d)Σ−1 for different values of c2 ranging from {0, 0.1, 0.5, 1} which controls for the
quality of the synthetic data. We run a small experiment with label noise levels σ1 = σ2 = 0.1,
input-dimension d = 600, number of real samples n1 = 300, and synthetic samples n2 = 200, for
a total of n = n1 + n2 = 500 samples. We fit a random projection model (cid:98)fRP according to (5) and
for different values of the width parameter m (to control the size the of the model), and report the
results in Figures 4 and 7. The regularization parameter λ is set to a very small value (10−8). We
also consider a variation of this experiment with different values of the synthetic dataset size n2 and
report the results in Figure 1.

B.2 TWO-LAYER NEURAL NETWORKS

The two-layer neural networks are trained using stochastic gradient descent (SGD) with a batch size
of 128 and a learning rate of 0.1. The models are trained for 400 epochs to fully converge. We
employ a held-out validation set from the training set to select the best checkpoint to evaluate.

B.3 LANGUAGE MODELING

The generation process for the BabiStories dataset is detailed in the GitHub repository of Zhang
et al. (2024a). The dataset comprises a training set of 2,200,000 stories and a validation set of
22,000 stories, created by prompting the Mistral-8x7B model. Each prompt includes a description
of the generation task, character names, specific words to be used, and the beginning of a story. The
dataset stores the beginnings of these stories along with their generated continuations.

In our experiments, we trained a GPT-2-small model on this dataset to generate synthetic data. The
model was trained using next-token prediction, utilizing the beginnings and continuations of stories
to have good story generation quality. To maintain consistency with the original prompt distribution,
we used all the prompts that were initially employed to generate BabiStories. During story genera-
tion, we applied a temperature setting of 1 with top-p decoding where p = 1. After generation, we
filtered out stories of poor quality, such as those containing unwanted symbols, following the same

17

102103104Hidden Dimension708090AccuracySynthetic size n2 = 2000102103104Hidden DimensionSynthetic size n2 = 10000102103104Hidden DimensionSynthetic size n2 = 40000Quality of Synthetic Data (Generator Performance)97.992.985.874.6Under review as a conference paper at ICLR 2025

procedure as in Zhang et al. (2024a). The filtered beginnings and synthetic continuations were then
collected to form the synthetic dataset.

We used a GPT-2 model with an embedding dimension of d = 768, 12 attention heads, and a context
length of 512 tokens, which typically encompasses one to three stories. During training, we applied
a learning rate of 5 × 10−3, a dropout rate of 0.05, L2 weight decay of 0.1, and a warm-up phase of
2,000 iterations.

C STATIC / SINGLE-STEP DATA MIXING

For the purposes of studying scaling laws for learning on mixtures of real and surrogate (e.g synthetic
data), the setting considered in (Jain et al., 2024) consists in the following optimization problem:

(cid:98)w = arg min
w∈Rn

α
n1

(cid:88)

(x⊤

i w − yi)2 +

(xi,yi)∈D1

(1 − α)
n2

(cid:88)

(xi,yi)∈D2

(x⊤

i w − yi)2 + λ∥w∥2.

(12)

This is an instance of weighted weighted empirical risk minimization (Shimodaira, 2000; Vogel
et al., 2021) where the weight the sample weight πi is constant across each group: πi = (1 −
α)n/n1 ≃ (1 − α)/p1 for real samples real samples vs πi = αn/n2 ≃ α/p2 for synthetic samples.
Thus α ∈ (0, 1) is a mixing coefficient for the two the two source of data; in particular α →
0 corresponds to only using real data (which corresponds to group 1) for training, while α →
1 corresponds to only using surrogate data (group 2). Formula (12) replaces the formula for the
weights vector (cid:98)w of the classical linear model (cid:98)fCL (3).
For conciseness, as in Section 5 we focus on the isotropic case considered in Section 3.1 where the
feature covariance matrices are Σ1 = Σ2 = Id and the shift matrix ∆ := cov(w∗
2) has the form
(c2/d)Id for some scalar c > 0. Further, let us consider the regime where d/n → 0 . In the language
of our paper, one should think of this as corresponding to the proportionate scaling regime given in
(4), and then letting ϕ → 0+ (extremely under-parametrized regime). We have the following result.

1 −w∗

(a) n1 = 1000, d = 500.

(b) n1 = 10000, d = 100, so that ϕ = d/n ≤ 100/10200 < 0.01 (small). Corollary 3 correctly
predicts that the optimal strategy mixing coefficient is α∗ ≈ 0, i.e to discard surrogate data altogether.

Figure 12: Failure of naive real+surrogate data mixing to solve model collapse. For this experiment, we
use different several different values for the size of the real data n1 and the synthetic data n2 . Solid curves
correspond to experiments while broken curves correspond to our theoretical prediction give in Corollary 3.
Error-bars correspond to independent runs.

Corollary 3. Consider the proportionate scaling limit (4). For small ϕ = d/n, it holds that
2α2 tr ∆ + ((1 − α)p1σ2

Etest( (cid:98)fCL) ≃ p2

2)ϕ + O(ϕ2).

1 + αp2σ2

(13)

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

The formula given in (13) represents a U-shaped function of α, minimized when α = α∗, with

(cid:18)

α∗ = clip[0,1]

1 −

p1σ2

1 − p2σ2
2
2 tr ∆

(cid:19)

ϕ

.

(14)

It should be clear that if tr ∆ = Ω(1) and σ1, σ2 = O(1), then α∗ → 0; this corresponds to only
using real data for training! In contrast any fixed value α ∈ [0, 1), leads a positive lower-bound
on test error Etest( (cid:98)fCL) ≳ tr ∆; this is effectively model collapse. The situation is empirically
confirmed in Figure 12.

D SOME OMITTED THEORETICAL RESULTS AND COMMENTS

D.1 CLASSICAL LINEAR MODEL IN OVER-PARAMETRIZED ISOTROPIC SETTING

We now complement the analysis presented at the end of Section 3.1 with an analysis for the case
ϕ ∈ (1, ∞). Plugging into Theorem 1 gives κ → ϕ − 1 and u → 1/(1 − ϕ/ϕ2) = ϕ/(ϕ − 1) in the
limit λ → 0+. We obtain the following corollary.
Corollary 4. For ϕ ∈ (1, ∞), in the limit (4) and λ → 0+, it holds that Etest ≃ E + ζ, with

E = V + B, B = r2(1 −

1
ϕ

), V =

σ2
ϕ − 1

,

ζ =

(cid:18)

p2

p2 c2
ϕ2

ϕ − p2
ϕ − 1

+ (ϕ − p2)2

(cid:19)

,

(15)

Moreover, for large ϕ ∈ (1, ∞), it holds that Etest( (cid:98)fCL) − E ≃ (1 − 2/ϕ) p2c2 + O(1/ϕ2).

Thus, for any fixed c > 0, strong model collapse occurs: the RHS vanishes only if p2 → 0+, i.e
only if we discard all but a vanishing proportion of synthetic data from the training dataset. This
is strong model collapse. Combining with Corollary 1, we conclude that (at least in the isotropic
setting, strong model collapse occurs both the under-parametrized and over-parametrized settings.

D.2 CONNECTIONS TO CLASSICAL MODEL COLLAPSE IN REGRESSION

In the setting of classical model collapse (Shumailov et al., 2023; 2024; Alemohammad et al., 2023;
Dohmatob et al., 2024b;a), we have w∗
ℓ Eℓ,where N is the number of iterations (i.e
self-loops) in the synthetic data-generation process. Let nℓ be the number of samples available for
2 + Eℓ) ∈ Rn×d × Rn, where the noise vectors Eℓ are
training at stage ℓ with training data (Xℓ, Xℓw∗
independent with iid components from N (0, σ2
ℓ ). In the proportionate scaling regime n1, . . . , nN →
∞ with d/nℓ → ϕℓ ∈ (0, ∞) for all ℓ, the situation is equivalent to taking

1 +(cid:80)N

ℓ=1 X †

2 = w∗

∆ =

(cid:88)

ℓ

ℓ · E (X †
σ2

ℓ )⊤X †

ℓ ≃

σ2
ℓ
nℓ − df 2(κℓ; Σ)

(cid:88)

ℓ

Σ(Σ + κℓId)−2, with κℓ = κ(nℓ, 0; Σ).

In particular, if maxℓ ϕℓ ≤ 1 (so that there is enough samples to perfectly fit the training data at each
stage of the iterative process), and for simplicity we set σℓ = σ0 for all ℓ, then the above expression
simplifies to ∆ ≃ (cid:0)(cid:80)
ℓ /(nℓ − d)(cid:1) Σ−1. More generally, consider the generic setting where
∆ ≃ (c2/d)Σ−1, for any c > 0, so that the previous setting corresponds to c2 = (cid:80)
ℓ ϕℓ/(1 − ϕℓ).
In the particular case where p1 → 0+, i.e only synthetic data is available for training. Theorem 1
then gives

ℓ σ2

ℓ σ2

· (cid:0)df 2 +uκ2 tr(Σ + κId)−2(cid:1) = η2 df 2 ·

(cid:18)
1 + κ2 df 2

n − df 2

tr(Σ + κId)−2

(cid:19)

.

ζ ≃

c2
d
In particular, taking c2 = (cid:80)

ℓ ϕℓ/(1 − ϕℓ) gives

ℓ σ2
(cid:18)
1 + κ2 df 2

n − df 2

ζ ≃

tr(Σ + κId)−2

(cid:19) df 2
d

σ2
ℓ ϕℓ
1 − ϕℓ

.

(cid:88)

ℓ

(16)

This is recovers the main result of (Dohmatob et al., 2024a).

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

E DETERMINISTIC EQUIVALENTS

Let Xj (resp. Yj) be the design matrix (resp. response vector) corresponding to dataset Dj. Thus, the
design matrix X1 ∈ Rn1×d for the real dataset has rows given by xi for i ∈ [n1] and Y1 ∈ Rn1 with
components yi for i ∈ [n1], with X2 ∈ Rn2×d and Y2 ∈ Rn2 defined analogously for the synthetic
dataset. Let X ∈ Rn×d(resp. Y ∈ Rn) be the design matrix (resp. response vector) corresponding
to the total dataset. We temporarily drop the condition Σ1 = Σ2 = Σ, and instead consider generally
different covariance matrices Σ1 and Σ2 for the marginal distribution of the features x under the real
data distribution P1 and the synthetic data distribution P2.

E.1 CLASSICAL LINEAR MODEL

Note that the weights (cid:98)w of the model (cid:98)fCL given in (3) can be written explicitly as (cid:98)w = RX ⊤Y ,
2 X2 + nλId)−1, a random matrix. Its test error
where R := (X ⊤X + nλId)−1 = (X ⊤
Etest( (cid:98)fCL) writes Etest( (cid:98)fCL) = EX,Y [( (cid:98)fCL(x) − x⊤w∗
. In Proposition 4,
we shall show that the RHS in the above can be decomposed into a sum of simply random quantities
of the form r(k)
(A, B) that we now describe and analyze.

1)2] = EX,Y ∥ (cid:98)w − w∗

(A) and r(k)

1 X1 + X ⊤

1∥2
Σ1

j

j

Let A and B be d×d positive-definite matrices with well-behaved spectral (this will be made precise
latter) and let λ > 0. In analyzing the bias-variance decomposition of the test error, we are ultimately
led to consider the following quantities

r(1)
j (A) := E tr AMj(M + λId)−1,

r(2)(A, B) := E tr A(M + λId)−1B(M + λId)−1,
r(3)
j (A, B) := E tr AMj(M + λId)−1B(M + λId)−1Mj,
r(4)
j (A, B) := E tr AMj(M + λId)−1B(M + λId)−1,

where we recall that M := M1 + M2 and Mj := X ⊤

j Xj/n.

Let (e1, e2) be the unique negative solution to the following pair of fixed-point equations

e1 =

1

1 + ϕ ¯tr Σ1K −1 , e2 =

1

1 + ϕ ¯tr Σ2K −1 , with K := p1e1Σ1 + p2e2Σ2 + λId.

Also, define (u1, u2) to be the unique positive solution to the pair of fixed-point equations

(17)

(18)

(19)

(20)

(21)

u1 = ϕe2
1

¯tr Σ1L′K −2, u2 = ϕe2
2

¯tr Σ2L′K −2, with L′ := p1u1Σ1 + p2u2Σ2 + λB.

(22)

Consider the following deterministic matrices

j (B + pj′uj′Σj′)Σ1 + u1(pj′ej′Σj′ + λId)2,

Cj := pje2
Dj := ejB − λujId + pj′(ejuj′ − ej′uj)Σj′,

where 1′ := 2 and 2′ = 1.

The following will be crucial for proving Theorem 1 and its corollaries.

Proposition 1. In the proportionate scaling limit (4), it holds for j = 1, 2 that

r(1)
j (A) ≃ pjej tr AΣjK −1,

r(2)(A, B) ≃ tr AL′K −2,
r(3)
j (A, B) ≃ pj tr AΣjCjK −2,
r(4)
j (A, B) ≃ pj tr AΣjDjK −2.

20

(23)

(24)

(25)

(26)

(27)

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

E.2 RANDOM PROJECTIONS

For d × d deterministic matrices A and B, define the following quenched quantities
r(3)
j (A, B) := E tr AMjSR⊤SBSRS⊤Mj,

r(1)
j (A) := E tr ASRS⊤Mj,

r(4)
j (A, B) := E tr AMjSR⊤SBSRS⊤,

r(5)(A, B) := E tr AM1SR⊤SBSRS⊤M2,

(28)

where we recall that R := (S⊤M S + λIm)−1, M := M1 + M2, Mj := X ⊤
will be useful because we may write

j Xj/n. These quantities

Vk =

σ2
j

1
n

(cid:88)

j

E tr MjSRS⊤ΣkSRS⊤ =

2
(cid:88)

j=1

σ2
j
n

r(4)
j (Id, Σk),

Bk = tr ΓΣk + E tr ΓM SRS⊤ΣSRS⊤M − 2 tr ΓΣkSRS⊤M + tr ∆M2SRS⊤ΣkSRS⊤M2

= tr ΓΣk + 2r(5)(Γ, Σk) + r(3)

1 (Γ, Σk) + r(3)

2 (Γ, Σk) − 2r(1)

1 (ΓΣk) − 2r(1)

2 (ΓΣk) + r(3)

2 (∆, Σk).

Each term in the above decomposition can now be analyzed via operator-valued free-probability
theory. The following proposition will be heavily exploited in the prove of Theorem 2.
Proposition 2. In the proportionate scaling limit (6), it holds that

r(1)
j (A) ≃ pjγτ ej tr AΣK −1,
r(4)
j (A, Σ) ≃ pjγ tr AΣDK −2,

r(3)
j (A, Σ) ≃ pj tr AΣCjK −2,
r(5)(A, Σ) ≃ p1p2γ tr AΣ2EK −2,

(29)

where the constants e1 and e2 and the matrices C1, C2, D, and E are as in Theorem 2.

E.3 PROOF OF PROPOSITION 1

WLOG, we only consider the case j = 1, and suppress this subscript henceforth from all the r(k)

j

’s.

j

Computing r(1)
. We only do j = 1 as j = 2 is completely analogous. One can obtain a minimal
9 × 9 linear pencil Q for the random matrix R = AM1(M + λId)−1 such that Q is a 9 × 9 block
matrix (not shown here1) and R = Q−1[1, 5]/λ (using zero-based indexing). It follows that in the
asymptotic limit, one has

r(1)/d = E ¯tr R ≃ G1,5,

(30)
where G = (id ⊗E ¯tr )[Q−1] ∈ M9(C) is the matrix containing the limiting expected values of the
normalized traces of the blocks of each of the 9 × 9 = 81 blocks of Q−1 (we define the trace of
each rectangular as zero). Using classical operator-valued free probability theory (OVFPT) Mingo
& Speicher (2017), we have the following fixed-point equations which define G1,5 implicitly

G1,5 = p1G3,3 ¯tr AΣ1(λId + p1G3,3Σ1 + p2G7,7Σ2)−1,

G3,3 =

G7,7 =

λ
λ − ϕG4,2
λ
λ − ϕG8,6

,

,

G4,2 = −λ ¯tr Σ1(λId + p1G3,3Σ1 + p2G7,7Σ2)−1,
G8,6 = −λ ¯tr Σ2(λId + p1G3,3Σ1 + p2G7,7Σ2)−1.

We deduce that G3,3 = e1, G7,7 = e2, and

r(1)/d = G1,5 = p1e1 ¯tr AΣ1(λId + p1e1Σ1 + p2e2Σ2)−1,

where (e1, e2) is the unique pair of nonnegative solutions to the system of equations

e1 =

e2 =

1
1 + ϕ ¯tr Σ1(λId + p1e1Σ1 + p2e2Σ2)−1 ,
1
1 + ϕ ¯tr Σ2(λId + p1e1Σ1 + p2e2Σ2)−1 .

1All the linear pencils in this work are very big and are omitted for brevity.

21

(31)

(32)

(33)

(34)

(35)

(36)

(37)

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

Putting things together gives

r(1) ≃ d · G1,5 = p1e1 tr AΣ1(p1e1Σ1 + p2e2Σ2 + λId)−1 = p1 tr AΣ1K −1.

In particular, in the limit p2 → 0+ (i.e single data source), the first equation becomes

1 − λ/κ1 = 1 − η1λ = ϕ1η1 ¯tr Σ1(Id + p1η1Σ1)−1

= ϕ1 ¯tr Σ1(κ1Id + Σ1)−1,

or equivalently,

κ1 − λ ≃ κ1

df 1(κ1; Σ1)
n1

.

Furthermore, r(1) is now given by

r(1) ≃ e1 tr AΣ1(e1Σ1 + λId)−1 = tr AΣ1(Σ1 + κ1Id)−1.

(38)

(39)

Computing r(4). Here, the minimal linear pencil for the random matrix involved R = AM1(M +
λId)−1B(M + λId)−1 is a 16 × 16 block matrix Q such that R = Q−1[1, 9]/λ. Thus, r(4)/d ≃
G1,16/λ, where G = (id ⊗E ¯tr )[Q−1] ∈ M16(C).
First consider the special case p2 → 0+ (i.e n2 is negligible compared to n1). The fixed-point
equations defining G1,9 are given by

G1,9 = λ ¯tr AΣ1(G3,3B + G3,11Id)(λId + G3,3Σ1)−1(λId + G11,11Σ1)−1,

G3,3 =

G11,11 =

,

λ
λ − ϕG4,2
λ
λ − ϕG12,10

,

G3,11 =

λϕG4,10
(λ − ϕG4,2)(λ − ϕG12,10)

=

ϕG3,3G11,11G4,10
λ

,

G12,10 = −λ ¯tr Σ1(λId + G11,11Σ1)−1,
G4,10 = −λ ¯tr Σ1(λB − G3,11Σ1)(λId + G3,3Σ1)−1(λId + G11,11Σ1)−1,
G4,2 = −λ ¯tr Σ1(λId + G3,3Σ1)−1.

(40)

(41)

(42)

(43)

(44)

(45)

(46)

Observe the equations for G3,11 and G4,10 further give G3,11 = −v, where v solves the equation
v = ϕG3,3G11,11 ¯tr Σ1(vΣ1 + λB)(λId + G3,3Σ1)−1(λId + G11,11Σ1)−1.

(47)

Now, let e be the unique non-negative solution to the equation

e =

1
1 + ϕ ¯tr Σ1(λId + eΣ1)−1 .

(48)

It is easy to see that we must have G3,3 = G11,11 = e and

r(4)/d =

G1,9
λ

= ¯tr AΣ1(eB − vId)(λId + Σ1)−2

= e−1 ¯tr ABΣ1(κId + Σ1)−2 − ve−2 ¯tr AΣ1(κId + Σ1)−2

(49)

vκ2
λ2
where κ := λ/e. Furthermore, v defined earlier now satisfies

¯tr ABΣ1(κId + Σ1)−2 −

κ
λ

=

¯tr AΣ1(κId + Σ1)−2,

v = ϕe2 ¯tr Σ1(vΣ1 + λB)(λId + eΣ1)−2
= ϕ ¯tr Σ1(vΣ1 + λB)(κId + Σ1)−1.

Solving for v gives

v =

ϕλ ¯tr BΣ1(κId + Σ1)−2
1 − ϕ ¯tr Σ2

1(κId + eΣ1)−2 ≃

λ tr BΣ1(κId + Σ1)−2
n − df 2(κ)

.

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

In particular, if B = Σ1 and A = Id, then

v =

λ df 2(κ)
n − df 2(κ)

,

and so we must have

r(4)/d =

G1,9
λ

=

=

=

=

≃

κ
λ
κ
λ
κ
λ
κ
λ
n
d

1
d
1
d
1
d

vκ2
λ2

¯tr Σ1(κId + Σ1)−2

¯tr Σ2

df 2(κ) −

1(κId + Σ1)−2 −
κ2
λ
κ
λ

1
d
1
d

df 2(κ) −

tr Σ1(κId + Σ1)−2 ·

· (df 1(κ) − df 2(κ)) ·

df 2(κ)
n − df 2(κ)
df 2(κ)
n − df 2(κ)

(50)

(n − df 1(κ)) ·

df 2(κ)
n − df 2(κ)

≃

1
ϕ

df 2(κ)
n − df 2(κ)
df 2(κ)
n − df 2(κ)

,

where, in the last 2 steps we have made use of the following identities which follow from the defini-
tion of κ

κ − λ ≃

κ df 1(κ)
n

,

κ tr Σ1(κId + Σ1)−2 = df 1(κ) − df 2(κ).

We deduce that the variance term in the bias-variance decomposition of the test error is given by

V ar = σ2 1
n

r(4) ≃ σ2 df 2(κ)

n − df 2(κ)

= σ2u = σ2 df 2(κ)/n

1 − df 2(κ)/n

.

(51)

Let us now compute the limiting value of r(4) for any values of the proportions p1, p2 ∈ (0, 1) with
p1 + p2 = 1. The fixed-point equations defining G1,9 now become
G1,9 = p1 ¯tr AΣ1S(λId + p1G2,2Σ1 + p2G6,6Σ2)−2,

with S := λ(G2,2B + G2,10Id) + p2(e2G2,10 − e1G6,13)Σ2,

G2,2 = e1,
G6,6 = e2,

G2,10 = G3,11 =

G6,13 = G7,14 =

ϕe2

ϕe2

1G4,10
λ
2G8,13
λ

,

,

G8,13 = λ ¯tr Σ2(λB − p1G3,11Σ1 − p2G7,14Σ2)(λId + p1G3,3Σ1 + p2G7,7Σ2)−2,
G4,10 = λ ¯tr Σ1(λB − p1G3,11Σ1 − p2G7,14Σ2)(λId + p1G3,3Σ1 + p2G7,7Σ2)−2,

where e1 ≥ 0 and e2 ≥ 0 solve the following system of equations

e1 =

e2 =

1
1 + ϕ ¯tr Σ1(λId + p1e1Σ1 + p2e2Σ2)−2 ,
1
1 + ϕ ¯tr Σ2(λId + p1e1Σ1 + p2e2Σ2)−2 .

(52)
(53)
(54)
(55)

(56)

(57)

(58)

(59)

(60)

(61)

Furthermore, we deduce that G6,13 = −v2 and G2,10 = −v1, where v1 and v2 solve the equations

v1 = ϕe2
1
v2 = ϕe2
2

¯tr Σ1(p1v1Σ1 + p2v2Σ2 + λB)(λId + p1e1Σ1 + p2e2Σ2)−2,
¯tr Σ2(p1v1Σ1 + p2v2Σ2 + λB)(λId + p1e1Σ1 + p2e2Σ2)−2.

(62)

(63)

Putting things together gives the formula for r(4) proposed in Proposition 1.

In particular, taking p2 → 0 (i.e p1 → 1) recovers the formula as a special case.

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Under review as a conference paper at ICLR 2025

Computing r(3). A minimal linear pencil for the corresponding random matrix R = AM1(M +
λId)−1B(M + λId)−1M1 is a 17 × 17 block matrix Q such that R = Q−1[1, 16]. This gives

where G = (id ⊗E ¯tr )[Q−1] ∈ M17(C). The fixed-point eqautions that determine G1,16 are

r(3)/d ≃ G1,16,

G1,16 = p1 ¯tr AΣ1S(λId + p1e1Σ1 + p2e2Σ2)−2
with S := p1e2
G7,14 = G6,13 = −v2,
G3,11 = G2,10 = −v1.

1(λB − p2G6,13Σ2)Σ1 − G2,10(λId + p2e2Σ2)2,

We deduce the formula given in Proposition 1. In particular, taking the limit p2 → 0 (i.e p1 → 1)
gives

1BΣ1 + λv1Id = e2

1BΣ1 + λ2u1Id,

• (cid:101)S ≃ e2
• v1 = ϕe2
1

¯tr Σ1(v1Σ1 + λB)(e1Σ1 + λId)−2 = ϕ ¯tr Σ(v1Σ1 + λB)(Σ + κ1Id)−2, i.e

u1 =

v1
λ

=

ϕ ¯tr BΣ1(Σ1 + κId)−2

1 − ϕ ¯tr Σ2

1(Σ1 + κ1Id)−2 ≃

tr BΣ1(Σ1 + κId)−2

n − df (1)

2 (κ1)

.

(64)

Finally, recalling that κ1 = λ/e1 by construction, we get

r(3) ≃ d · G1,16 = e2

1 tr ABΣ2

1(e1Σ1 + λId)−2 + λ2u1 ¯tr AΣ1(e1Σ1 + λId)−2

= tr ABΣ2

1(Σ1 + κ1Id)−2 +

λ2u1
e2
1

tr AΣ1(Σ1 + κ1Id)−2

≃ tr ABΣ2

1(Σ1 + κ1Id)−2 + κ2

1 tr AΣ1(Σ1 + κ1Id)−2 ·

tr BΣ1(Σ1 + κId)−2

n − df (1)

2 (κ1)

.

Computing r(2). A pencil for the relevant matrix R = λ2A(M + λId)−1B(M + λId)−1 has min-
imal linear pencil Q of size 15 × 15, where R = Q−1[1, 8]. We deduce that r(2)/d = E ¯tr R/λ2 =
G1,8/λ2, where G = (id ⊗E ¯tr )Q−1 ∈ M15(C). The fixed-point equations defining G1,8 are given
by

G1,8 = λ ¯tr AS(p1G2,2Σ1 + p2G5,5Σ2 + λId)−2,

with S = λB − p1G2,9Σ1 − p2G5,12Σ2,

G2,2 = e1,
G5,5 = e2,

G2,9 = G3,10 =

G5,12 = G6,13 =

,

ϕe2
1G4,9
λ
ϕe2
2G7,12
λ

,

G4,9 = −λ ¯tr Σ1(λB − p1G3,10Σ1 − p2G6,13Σ2)(p1G2,2Σ1 + p2G5,5Σ2 + λId)−2,
G7,12 = −λ ¯tr Σ2(λB − p1G3,10Σ1 − p2G6,13Σ2)(p1G2,2Σ1 + p2G5,5Σ2 + λId)−2.

(65)
(66)
(67)
(68)

(69)

(70)

(71)

(72)

Based on previous calculations, we deduce that G2,9 = −v1 and G5,12 = −v2, and so

r(2) ≃ d ·

G1,8
λ2 =

1
λ

tr A(p1v1Σ1 + p2v2Σ2 + λB)(p1e1Σ1 + p2e2Σ2 + λId)−2 = tr A(cid:101)LK −2,

as claimed. This completes the proof of Proposition 1.

E.4 PROOF OF PROPOSITION 2

In Section G.1 we will establish a more general result which implies Proposition 2 as a special case.

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

F PROOF OF THEOREM 1 AND COROLLARIES

Let us note that the results in Bach (2023) were obtained in a two-stage approach, where random
matrix theory is applied on the raw (unquenched test error ∥ (cid:98)w − w1∥2
Σ with the projection matrix
treated like a deterministic matrix, and then RMT is done one more on the resulting expressions but
now treating S as random. The case general case p2 ∈ (0, 1) is much more difficult; the key technical
difficulty can be pinned down to the problem of obtaining analytic deterministic equivalents for the
trace of the and derivatives of the resolvent of a sum of random matrices. To circumvent this, we
employ the tools of operator-valued free probability theory.

F.1 PROOF OF THEOREM 1

From Proposition 4 and 1 applied with Σ1 = Σ2 = Σ, we know that

Etest( (cid:98)fCL) = V1 + B1, with

V1 ≃

2
(cid:88)

j=1

pjσ2
j

1
n

tr ΣkDj,kK −2 =

2
(cid:88)

j=1

pjσ2
j

κ
λ

·

1
n

= σ2 κ
λ

·

1
n

tr Σ(Σ − κuId)(Σ + κId)−2,

B1 = p2 tr ∆Σ2C2,1K −2 + λ2 tr ΓL′

1K −2

tr Σ(Σ − κuId)(Σ + κId)−2

= p2 tr ∆Σ (cid:0)p2(1 + p1u)Σ2 + u(p1Σ + κId)2(cid:1) (Σ + κId)−2 + κ2(u + 1) tr ΓΣ(Σ + κId)−2.

Now, for the V1 term, first observe that

tr Σ(Σ − κuId)(Σ + κId)−2 = tr Σ(Σ −

κ df 2
n − df 2

Id)(Σ + κId)−2

= df 2 −

= df 2 −

df 2
n − df 2
df 2
n − df 2

· κ tr Σ(Σ + κId)−2

(df 1 − df 2)

=

df 2
n − df 2

(n − df 1).

We deduce that

V1 = σ2 · (1 − df 1 /n)

κ
λ

·

df 2
n − df 2

= σ2 ·

df 2
n − df 2

=: V,

where we have used the identity κ − λ = κ df 1 /n, which defines κ.

We now handle the B1 term. First observe that u + 1 = n/(n − df 2), and so one computes

κ2(u + 1) tr ΓΣ(Σ + κId)−2 = κ2

n
n − df 2

tr ΓΣ(Σ + λId)−2 =: B,

which is the classical formula for the bias term.

To finalize, observe that

tr ∆ΣC2,1K −2 = tr ∆Σ (cid:0)p2(1 + p1u)Σ2 + u(p1Σ + κId)2(cid:1) (Σ + κId)−2

= p2(1 + p1u) tr ∆Σ3(Σ + κId)−2 + u tr ∆Σ(p1Σ + κId)2(Σ + κId)−2 =: ζ,

which concludes the proof.

25

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Under review as a conference paper at ICLR 2025

F.2 PROOF OF COROLLARY 1

Indeed, here we have κ → 0 and u → ϕ/(1 − ϕ) in the limit λ → 0+. Theorem 1 then gives

Etest( (cid:98)fCL) ≃ V + B + ζ, where V =

, B = 0,

σ2ϕ
1 − ϕ
(cid:0)p2(1 − ϕ + p1ϕ) + p2

1ϕ(cid:1) =

(p2 + (p1 − p2)ϕ) = p2

2c2 +

ζ =

=

p2c2
1 − ϕ
p2c2
1 − ϕ

(p2(1 − p2ϕ) + p2

1ϕ)

p2c2
1 − ϕ
p2p1c2ϕ
1 − ϕ

.

For small ϕ, this further gives Etest( (cid:98)fCL) ≃ σ2ϕ/(1 − ϕ) + p2
O(ϕ2).

2c2 + O(ϕ2) ≃ σ2d/n + p2

2c2 +

F.3 PROOF OF COROLLARY 3

The setup can be seen as a special instance of the setup considered in the proof of Proposition 1 (cf.
Appendix F.1), since it corresponds to taking Σ1 = (1 − α)Σ/p1, and Σ2 = αΣ/p2. We must have

1
e1
1
e2

= 1 + ϕ ¯tr Σ1K −1 = 1 +

= 1 + ϕ ¯tr Σ2K −1 = 1 +

(1 − α)ϕ/p1
(1 − α)e1 + αe2 + λ
αϕ/p2
(1 − α)e1 + αe2 + λ

,

.

(73)

(74)

At least for λ = 0 and 0 < ϕ < 1, these equations can be solved explicitly to get e1, e1 ≥ 0 but the
resulting formulae are rather complicated, and therefore are omitted altogether. In any case, heorem
1 correctly predicts the test error, as can be seen in Figure 12.
A particular case where things are solvable to give simple expressions, is when ϕ → 0+. In this
limit, it is easy to see that e1 = e2 = 1 and u1 = u2 = 0. This gives

K = Σ + λId,
L′ = Sigma,
C1 = (1 − α)Σ,
C2 = αΣ,
Dk = Σ,

λ2r(2)(A, Σ) ≃ λ2 tr AΣ(Σ + λId)−2 = λ · (cid:0)tr AΣ(Σ + λ)−1 − tr AΣ2(Σ + λId)−2(cid:1) ,

r(3)
1 (A, Σ) ≃ p1 tr AΣ1C1K −2 = p1
r(3)
2 (A, Σ) ≃ p2 tr AΣ2C2K −2 = p2
r(4)
1 (A, Σ) ≃ p1 tr AΣ1D1K −2 = p1α tr AΣ2(Σ + λId)−2,
r(4)
2 (A, Σ) ≃ p2 tr AΣ2D2K −2 = p2(1 − α) tr AΣ2(Σ + λId)−2.

1α2 tr AΣ2(Σ + λId)−2,
2(1 − α)2 tr AΣ2(Σ + λId)−2,

We deduce that

V1 =

2
(cid:88)

j=1

pj

σ2
j
n

j (Id, Σ) = (cid:0)(1 − α)p1σ2
r(4)

1 + αp2σ2
2

(cid:1) df 2(λ; Σ)
n

,

B1 = r(3)

2 (∆, Σ) + λ2r(2)(Γ, Σ) λ→0+

−→ p2

2(1 − α)2 tr ∆.

(75)

(76)
(77)
(78)
(79)

(80)

(81)

(82)

(83)

(84)

(85)

(86)

Putting things together then gives

Etest( (cid:98)fCL) ≃ B1 + V1 ≃ p2

2(1 − α)2 tr ∆ + (αp1σ2

1 + (1 − α)p2σ2
2)

d
n

,

as claimed.

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

F.4 PROOF OF COROLLARY 2

Applying the first part of Corollary 1 recursively gives for any iteration t ≥ 1,

Etest( (cid:98)f (t)

CL) ≃ c2

t ≃ E + p2

2c2

t−1 ≃ . . . ≃ p2t

2 c2

0 +

1 − p2t
2
1 − p2
2

E, with E :=

σ2ϕ
1 − ϕ

.

Iterating the above gives

c2
t+1 ≃

σ2ϕt
1 − ϕt

+ p2

2c2

t , ϕt = d/Nt, Nt = n,

0 = c2.
c2

(87)

Setting E := σ2ϕ/(1 − ϕ) ≃ σ2d/n, we get

Etest( (cid:98)f (t+1)

CL ) ≃ c2

t+1 ≃ p2

σ2ϕt
1 − ϕt

2c2
t +
(cid:18)

≃ p2
2

2c2
p2

t−1 +

σ2ϕt−1
1 − ϕt−1

(cid:19)

+

σ2ϕt
1 − ϕt
σ2ϕt
1 − ϕt

+

≃ p2(1+1)
2

t−1 + p2
c2
2

σ2ϕt−1
1 − ϕt−1

...

≃ p2(t+1)
2

c2
0 +

(cid:88)

0≤j≤t

σ2ϕj
1 − ϕj

p2(t−j)
2

= p2(t+1)
2

c2 + E

(cid:88)

p2j
2

= p2(t+1)
2

c2 +

0≤j≤t
1 − p2(t+1)
2
1 − p2
2

E.

In particular, we if p2 is bounded away from 1 (i.e if p1 := 1 − p2 = Ω(1)), we get

Etest( (cid:98)f (t)

CL) ≃

1
1 − p2
2

E + p2t

2 c2,

for large t. The first part is just a constant multiple of the scaling law we would have with training
on a dataset comprising of n units of clean data.

On the other hand, we have

lim
p2→1

Etest( (cid:98)f (t)

CL) ≃ c2 + tE.

This is an increasing function of t, lower-bounded by c2 + E. We recover the classical picture, in
which model collapse prevails (depending on the size of c2, as per Corollary 1).

F.5 PROOF OF COROLLARY 4

From Theorem 1 and the observation that κ → ϕ − 1 and u → 1/(1 − ϕ/ϕ2) = ϕ/(ϕ − 1) in the
limit λ → 0+, we have Etest( (cid:98)w) ≃ E + ζ, with

E = V + B, B = r2 (ϕ − 1)2

ϕ2

1
1 − 1/ϕ

= r2 (1 − 1/ϕ) ,

V =

σ2
ϕ − 1

,

ζ =

(cid:18)

p2 c2
ϕ2

p2(1 +

p1
ϕ − 1

) + (p1 + ϕ − 1)2

(cid:19)

,

and the first part of the result follows after some algebra.

The second part then follows from expanding the above around ϕ = ∞.

27

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Under review as a conference paper at ICLR 2025

G PROOF OF PROPOSITION 2 AND THEOREM 2

G.1 PROOF OF PROPOSITION 2

We state and proof a more general result without the requirement Σ1 = Σ2 = Σ.

Let (e1, e2, τ ) be the unique nonnegative solution to the following fixed-point equations
1
1 + ψτ ¯tr Σ1K −1 ,
1
1 + ψτ ¯tr Σ2K −1 ,
1
1 + ¯tr K0K −1 ,

e2 =

e1 =

τ =

with K0 := p1e1Σ1 + p2e2Σ2, K := γτ K0 + λId.

(88)

(89)

(90)

(91)

Also, let (v1, v2, ω) to be the unique nonnegative solution to the following fixed-point equations

v1 = ψe2
1
v2 = ψe2
2
ω = τ 2 ¯tr (γK 2

¯tr Σ1(γτ 2L + λωId)K −2,
¯tr Σ2(γτ 2L + λωId)K −2,
0 + λL)K −2,

with L := p1v1Σ1 + p2v2Σ2 + λB.

Finally, define d × d matrices C1, C2, D1, D2, E by

(cid:0)γτ 2(B + p2u2Σ2) + ωId
(cid:0)γτ 2(B + p1u1Σ1) + ωId

C1 := γp1e2
1
C2 := γp2e2
2
D1 := τ 2e1B + (e1ω − τ v1)Id + γτ 2p2(e1u2 − e2u1)Σ2,
D2 := τ 2e2B + (e2ω − τ v2)Id + γτ 2p1(e2u1 − e1u2)Σ1,
E := γ(γτ 2B + ωId),

(cid:1)Σ1 + u1(γτ p2e2Σ2 + λId)2,
(cid:1)Σ2 + u2(γτ p1e1Σ1 + λId)2,

Proposition 3. In the proportionate scaling limit (6), it holds that
r(1)
j (A) ≃ γτ pjej tr AΣjK −1,

r(3)
j (A, B) ≃ γpjAΣjCjK −2,
r(4)
j (A, B) ≃ γpj tr AΣjDjK −2,
r(5)(A, B) ≃ tr AEK −2.

(92)

(93)

(94)
(95)

(96)

(97)

(98)

(99)

(100)

(101)

(102)

(103)

(104)

Observe that if we force τ = γ = 1 and ω = 0, then we recover the corresponding formulae given
in Proposition 1. On the other hand, taking Σ1 = Σ2 = Σ gives Proposition 2.

Proof. WLOG, we only consider the cases where j = 1.

Computing r(1)
(zero-based indexing). We deduce that

1 . There is a 11 × 11 minimal linear pencil Q such that ASRS⊤M1 = Q−1[1, 10]

:= E tr ASRS⊤M1 ≃ d · G1,10,
(105)
where G := (id ⊗E ¯tr )Q−1 ∈ C11×11. Moreover, G1,10 is given by the following fixed-point
equations

r(1)
1

G1,10 = p1γG2,2G5,5 ¯tr AΣ1K −1,

with K := γG2,2L + λId, L := p1G5,5Σ1 + p2G8,8Σ2,

G5,5 =

G8,8 =

1
1 + ϕγG2,2 ¯tr Σ1K −1 =
1
1 + ϕγG2,2 ¯tr Σ2K −1 =

1
1 + ψG2,2 ¯tr Σ1K −1 ,
1
1 + ψG2,2 ¯tr Σ2K −1 ,

G2,2 =

1
1 + ¯tr LK −1 ,

28

(106)
(107)

(108)

(109)

(110)

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

(113)

(114)
(115)

(116)

(117)

(118)

(119)

(120)

Under review as a conference paper at ICLR 2025

Then, one deduces that

tr ASRS⊤M1 ≃ d · G1,10 = p1e1τ γ tr AΣ1K −1.

(111)

Computing r(4)
deduce that

1 . Here, the pencil Q is 20 × 20 and AM1SRS⊤SRS⊤ = −Q−1[1, 13]/λ. We

:= E tr AM1SRS⊤BSRS⊤ ≃ −d · G1,13/λ,
(112)
where G := (id ⊗E ¯tr )Q−1 ∈ C20×20. Moreover, G1,13 is given by the following fixed-point
equations

r(4)
1

−G1,13 = p1γ ¯tr AΣ1T K −2, where

T := λ(τ 2e1B + (e1G6,12 + τ G3,15)Id) + p2γτ 2(e2G3,15 − e1G9,18)Σ2,

G12,12 = G6,6 = τ,
1G4,14
λ

G3,15 =

ϕe2

G4,14 = −λγ ¯tr Σ1
2G10,17
λ

G9,18 =

ϕe2

G10,17 = −λγ ¯tr Σ2
G6,12 = −τ 2G7,11,
G7,11 = − ¯tr (γK 2

,
(cid:0)γτ 2(p1G3,15Σ1 + p2G9,18Σ2) − λ(γτ 2B + G6,12Id)(cid:1) K −2,

,
(cid:0)γτ 2(p1G3,15Σ1 + p2G9,18Σ2) − λ(γτ 2B + G6,12Id)(cid:1) K −2,

0 + λ(λB − p1G3,15Σ1 − p2G9,18Σ2))K −2,

(121)
We deduce that G3,15 = −v1, G9,18 = −v2, and G6,12 = ω, where v1, v2, ω ≥ 0 solve the
following fixed-point equations
¯tr Σ1
v1 = ϕγe2
1
¯tr Σ1(γτ 2L + λωId)K −2,
= ψe2
1
¯tr Σ2
v2 = ϕγe2
2
¯tr Σ2(γτ 2L + λωId)K −2,
= ψe2
2
ω = τ 2 ¯tr (γK 2

(cid:0)γτ 2(p1v1Σ1 + p2v2Σ2) + λ(γτ 2B + ωId)(cid:1) K −2

(cid:0)γτ 2(p1v1Σ1 + p2v2Σ2) + λ(γτ 2B + ωId)(cid:1) K −2

0 + λ(λB + p1v1Σ1 + p2v2Σ2))K −2 = τ 2 ¯tr (γK 2

0 + λL)K −2,

with L := p1v1Σ1 + p1v2Σ2 + λB. Putting everything together then gives

r(4)
j ≃ −

d · G1,13
λ

= p1γ tr AΣ1 (cid:101)T K −2, where

(cid:101)T := T /λ = τ 2e1B + (e1ω − τ v1)Id + p2γτ 2(e1u2 − e2u1)Σ2 =: D1.

1 . The matrix of interest AM1SRS⊤BSRS⊤M1 admits a minimal linear pencil Q

Computing r(3)
of size 21 × 21, such that the formal equals to Q−1[1, 20]. It follows that
:= E tr AM1SRS⊤BSRS⊤M1 ≃ d · G1,20,

(122)

r(3)
1

where G := (id ⊗E ¯tr )Q−1 ∈ C21×21. The fixed-point equations defining G1,20 are

G1,20 = p1 ¯tr AΣ1(T /λ)K −2, where

T := p1G2

3,3γ(γτ 2(λB − p2G9,18Σ2) + λG6,12Id)Σ1 − G3,15(γτ p2G9,9Σ2 + λId)2,

G3,3 = e1,
G9,9 = e2,
G6,12 = ω,
G3,15 = −v1,
G9,18 = −v2.

Putting things together gives

r(3)
1 ≃ d · G1,20 = tr AΣ1 (cid:101)T K −2,
where (cid:101)T := T /λ = γp1e2
1

(cid:0)γτ 2(B + p2u2Σ2) + ωId

which completes the proof.

(cid:1)Σ1 + u1(γτ p2e2Σ2 + λId)2 =: C1,

29

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Under review as a conference paper at ICLR 2025

G.2 RECOVERING THEOREM 1 FROM THEOREM 2

Indeed, we have

ω′ → 0, θ → κ,

u →

ϕI2,2(κ)
1 − ϕI2,2(κ)

=

df 2(κ)/n
1 − df 2(κ)/n

,

for any regularization strength λ > 0, where κ is as defined in equation (7). Refer to Lemma 1.
Plugging these limits into the formulae provided in Theorem 2 then recovers Theorem 1.

G.3 PROOF OF THEOREM 2

This follows directly from Proposition 2 and the computations in Section I.2.

H PHASE-DIAGRAM FOR RANDOM PROJECTIONS MODEL

H.1 THE GENERAL REGULARIZED CASE

Lemma 1. The scalars u and ω′ which appear in Theorem 2, and described in Definition 2, solve
the following pair of linear equations

(123)

(124)

(125)

(126)

u = ϕI2,2(θ)(1 + u) + ϕI1,2(θ)ω′,
γω′ = I2,2(θ)ω′ + θ2I1,2(θ)(1 + u).

Furthermore, the solutions can be explicitly represented as

u =

ϕz
γ − ϕz − I2,2(θ)

, ω′ =

θ2I2,2(θ)
γ − ϕz − I2,2(θ)

,

where z = I2,2(θ)(γ − I2,2(θ)) + θ2I1,2(θ)2.

In particular, in the limit γ → ∞, it holds that

θ ≃ κ, ω′ → 0,

u ≃

ϕI2,2(κ)
1 − ϕI2,2(κ)

≃

df 2(κ)/n
1 − df 2(κ)/n

,

where κ > 0 is as defined in (7).

Proof. The equations defining these are

u = ψe2 ¯tr Σ(γτ 2L′ + ωId)K −2,
ω = τ 2 ¯tr (γωK 2
where K0 = eΣ, K = γτ K0 + λId, and L′
L′ = (1 + u)Σ. Now, we can rewrite the previous equations like so

0 + λ2L′)K −2,
:= uΣ + B. Further, since B = Σ, we have

(127)

u = ψe2 ¯tr Σ(γτ 2(1 + u)Σ + ωId)K −2 = ϕγ2τ 2e2(1 + u) ¯tr Σ2K −2 + ϕγe2ω ¯tr ΣK −2,
ω = τ 2 ¯tr (γωe2Σ2 + λ2(1 + u)Σ)K −2 = γτ 2e2ω ¯tr Σ2K −2 + λ2τ 2(1 + u) ¯tr ΣK −2.

This can be equivalently written as

u = ϕ(1 + u)γ2τ 2e2 ¯tr Σ2K −2 + ϕω′γ2τ 2e2 ¯tr ΣK −2,

γω′ = ω′γ2τ 2e2 ¯tr Σ2K −2 + (1 + u)λ2 ¯tr ΣK −2.

Now, observe that

τ 2e2 ¯tr Σ2K −2 = ¯tr Σ2(Σ + θId)−2/γ2 = I2,2(θ)/γ2,
τ 2e2 ¯tr ΣK −2 = ¯tr Σ(Σ + θId)−2/γ2 = I1,2(θ)/γ2,
λ2 ¯tr ΣK −2 = θ2 ¯tr Σ(Σ + θId)−2 = θ2I1,2(θ),
e2 ¯tr ΣK −2 = ¯tr Σ(Σ + θId)−2/(γτ )2 = I1,2(θ)/(γτ )2,
τ 2 ¯tr ΣK −2 = ¯tr Σ(Σ + θId)−2/(γe)2 = I1,2(θ)/(γe)2,

(128)

(129)

(130)

(131)

(132)

(133)

(134)

30

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

where we have used the definition θ = λ/(γτ e). Thus, u and ω have limiting values u and ω
respectively, which solve the system of linear equations

u = ψγ · γ−2I2,2(θ)(1 + u) + ψγ · γ−2I1,2ω′ = ϕI2,2(θ)(1 + u) + ϕI1,2(θ)ω′,

γω′ = I2,2(θ)ω′ + θ2I1,2(θ)(1 + u) = I2,2(θ)ω′ + θ2I1,2(θ)(1 + u),

where we have used the identity ϕγ = ψ. These correspond exactly to the equations given in the
lemma. This proves the first part.

For the second part, indeed, τ = 1 − η0/γ → 1 in the limit γ → ∞, and so θ ≃ λ/(γe) which
verifies the equation

θ ≃ λ + λψ ¯tr Σ(γeΣ + λ)−1 = λ + ϕ ·

λ
γe

¯tr Σ(Σ +

λ
γe

Id)−1 ≃ λ + θ tr Σ(Σ + θId)−1/n,

i.e θ ≃ λ + θ df 1(θ)/n and θ > 0. By comparing with the equation κ − λ = κ df 1(κ)/n satisfied
by κ > 0 in (7), we conclude θ ≃ κ.
Now, the equations (123) become ω′ = 0, and u = ϕI2,2(κ)(1 + u), i.e

u =

ϕI2,2(κ)
1 − ϕI2,2(κ)

≃

df 2(κ)/n
1 − df 2(κ)/n

,

as claimed.

H.2 UNREGULARIZED LIMIT

Define the following auxiiliary quantities

θ :=

λ
γτ e

, χ :=

λ
τ

,

κ :=

λ
e

.

where τ , e, u, and ω are as previously defined in Section 3.2.
Lemma 2. In the limit λ → 0+, we have the following analytic formulae

χ → χ0 = (1 − ψ)+ · γθ0,
κ → κ0 = (ψ − 1)+ · θ0/ϕ,
τ → τ0 = 1 − η0/γ,
e → e0 = 1 − ϕη0.

(135)

(136)
(137)
(138)
(139)

Proof. From equations (8) and the constraint Σ1 = Σ2 = Σ, we know that e1 = e2 = e, where e
and τ are unique positive solutions to a pair of fixed point equations. Observe that K0 = eΣ and
K = γτ K0 + λId = γτ e · (Σ + θId). Defining η := I1,1(θ), one can then rewrite the equations
defining e and τ as follows

e′ =

τ ′ =

λ
e
λ
τ

= λ + ψτ λ ¯tr ΣK −1 = λ +

ψτ λ
γτ e

¯tr Σ(Σ + θId)−1 = λ + ϕηe′,

= λ + λ ¯tr K0K −1 = λ +

λe
γτ e

¯tr Σ(Σ + θId)−1 = λ + (η/γ)τ ′.

We deduce that

e′ =

λ
1 − ϕη

,

τ ′ =

λ
1 − η/γ

,

τ ′e′ = λγθ.

(140)

(141)

(142)

In particular, the above means that η ≤ min(γ, 1/ϕ). The last part of equations (142) can be
rewritten as follows

λ
(1 − ϕη)(1 − η/γ)

= γθ, i.e ϕη2 − (ϕγ + 1)η + γ −

λ
θ

= 0.

(143)

31

Under review as a conference paper at ICLR 2025

This is a quadratic equation for η as a function of λ and θ, with roots

η± =

ϕγ + 1 ± (cid:112)(ϕγ + 1)2 − 4(ϕγ − (ϕ/θ)λ)
2ϕ

ψ + 1 ± (cid:112)(ψ + 1)2 − 4(ψ − ϕ/θ′)
2ϕ

=

. (144)

Now, for small λ > 0 and ψ ̸= 1, we can do a Taylor expansion to get

More explicitly,

η± ≃

ψ + 1 ± |ψ − 1|
2ϕ

±

1
θ|ψ − 1|

λ + O(λ2).

η+ ≃ O(λ2) +

η− ≃ O(λ2) +

(cid:26)1/ϕ + λ/((1 − ψ)θ),
γ + λ/((ψ − 1)θ),
(cid:26)γ − λ/((1 − ψ)θ),

1/ϕ − λ/((ψ − 1)θ),

if ψ < 1,
if ψ > 1.

if ψ < 1,
if ψ > 1,

Because η ≤ min(1, 1/ϕ, γ), we must have the expansion

η ≃ O(λ2) +

= η0 −

(cid:26)γ − λ/((1 − ψ)θ),

1/ϕ + λ/((ψ − 1)θ),
1
(1 − ψ)θ0

λ + O(λ2),

if ψ < 1,
if ψ > 1,

(145)

provided θ0 > 0, i.e η0 ̸= 1. in this regime, we obtain

τ ′ =

λ
1 − η/γ

e′ =

λ
1 − ϕη

≃

≃

(cid:26)λ/(1 − 1 + λ/((1 − ψ)γθ0)) = (1 − ψ)γθ0,

λ/(1 − 1/ψ + o(1)) → 0,

(cid:26)λ/(1 − ψ + o(1)) → 0,

λ/(1 − 1 + λϕ/((ψ − 1)θ0) → (ψ − 1)θ0/ϕ,

if ψ ≤ 1,
if ψ > 1,

if ψ ≤ 1,
if ψ > 1,

τ = 1 − η/γ ≃ 1 − η0/γ = (1 − 1/ψ)+,
e = 1 − ϕη ≃ 1 − ϕη0 = (1 − ψ)+.

On the other hand, if θ0 = 0 (which only happens if ψ < 1 and γ > 1 OR ψ ≥ 1 and ϕ ≤ 1), it is
easy to see from (142) that we must have τ ′ → 0, e′ → 0, τ → 1 − 1/γ, e → 1 − ϕ ≥ 0.
Next, let’s compute the limiting values of u and ω′ := ω/τ 2.

I RAW BIAS-VARIANCE DECOMPOSITION

I.1 CLASSICAL LINEAR MODEL

Proposition 4. Evaluated on the distribution Pk = PΣk,σ2
in (3) is given by

k,w∗
k

, the test error of model (cid:98)fCL defined

Etest( (cid:98)fCL) = Bk + Vk,

where Vk =

2
(cid:88)

σ2
j
n

r(4)
j (Id, Σk),

j=1
(cid:40)

Bk =

r(3)
2 (∆, Σ1) + λ2r(2)(Γ, Σ1),
1 (∆, Σ2) + λ2r(2)(Γ + ∆, Σ2) + 2λr(4)
r(3)

1 (∆, Σ2),

(146)

(147)

(148)

if k = 1,
if k = 2.

32

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Under review as a conference paper at ICLR 2025

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

Proof. Indeed, one computes
k∥2
Σk

ED∥ (cid:98)w − w∗

k)2]

k∥2
Σk

(cid:98)w − x⊤w∗

Ex∼N (0,Σk)[(x⊤

= EX1,Y1,X2,Y2
= EX1,Y1,X2,Y2 ∥ (cid:98)w − w∗
= EX1,Y1,X2,Y2 ∥(M + λId)−1X ⊤Y /n − w∗
= EX1,Y1,X2,Y2 ∥(M + λId)−1X ⊤(X1w∗
= EX1,Y1,X2,Y2 ∥(M + λId)−1(M1w∗
= Bk + Vk,1 + Vk,2.

k∥2
Σk
1 + E1, X2w∗
2) − w∗

1 + M2w∗

2 + E2)/n − w∗
k∥2
+ V1 + V2
Σk

k∥2
Σk

where

Bk := E ∥(M + λId)−1(Mkw∗

k + M−kw∗

−k) − w∗

Vk,j :=

σ2
j
n

E tr Mj(M + λId)−1Σk(M + λId)−1 =

r(4)
j (Id, Σk).

k∥2
,
Σk
σ2
j
n

(149)

(150)

It remains to analyze the bias term Bk. To this end, observe that
(M + λId)−1Mk = Id − (M + λId)−1(M−k + λId) = Id − (M + λM )−1M−k − λ(M + λId)−1.
Denoting M−1 = M2, M−2 = M1, w∗
2 −w∗
2, w∗
1,
we deduce that

1, and δk = (−1)kδ, where δ := w∗

−1 = w∗

−2 = w∗

k + (M + λId)−1M−kw∗

(M + λId)−1Mkw∗
= (M + λId)−1M−kw∗
= −(M + λId)−1M−kδk − λ(M + λId)−1w⋆
k.

−k − w∗
k
k − λ(M + λId)−1w∗
−k − (M + λId)−1M−kw∗
k

Since w∗
tively, we deduce that

1 and δ1 = δ := w∗

2 − w∗

1are independent with distributions N (0, Γ) and N (0, ∆) respec-

B1 = ∥(M + λId)−1M2δ − λ(M + λId)−1w⋆

1∥2
Σ1
= tr ∆M2(M + λId)−1Σ1(M + λId)−1M2 + λ2 tr Γ1(M + λId)−1Σ1(M + λId)−1
= r(3)

2 (∆, Σ1) + λ2r(2)(Γ, Σ1).
On the other hand, we have B2 = B2,1 + B2,2, where

2∥2
B2 = ∥ − (M + λId)−1M1δ − λ(M + λId)−1w⋆
Σ2
= ∥ − (M + λId)−1M1δ − λ(M + λId)−1(w⋆
1 + δ)∥2
Σ2
= ∥ − (M + λId)−1 (M1 + λId) δ − λ(M + λId)−1w⋆
1∥2
Σ2
= tr ∆(M1 + λId)(M + λId)−1Σ2(M + λId)−1(M1 + λId) + λ2 tr Γ(M + λId)−1Σ2(M + λId)−1
= tr ∆M1(M + λId)−1Σ2(M + λId)−1M1 + λ2 tr ∆(M + λId)−1Σ2(M + λId)−1
+ 2λ tr ∆M1(M + λId)−1Σ2(M + λId)−1 + λ2 tr Γ(M + λId)−1Σ2(M + λId)−1
1 (∆, Σ2) + λ2r(2)(Γ + ∆, Σ2) + 2λr(4)

1 (∆, Σ2).

= r(3)

This completes the proof.

I.2 RANDOM PROJECTIONS MODEL

We now expand the test error Etest( (cid:98)fRP ) of the random projections model (cid:98)fRP defined in (5). For
convenience, we recall the definition of the model here. Let S be a d × m random matrix with iid
(cid:98)v, where Φ(x) := S⊤x ∈
entries from N (0, 1/d). The model (cid:98)fRP is defined by (cid:98)fRP (x) := Φ(x)⊤
Rm defines a random feature map, and (cid:98)v ∈ Rm is given by

arg min
v∈Rm

L(w) =

∥Φ(Xk)v − Yk∥2
2
n

(cid:88)

k

+ λ∥v∥2
2.

(151)

33

Under review as a conference paper at ICLR 2025

Note that the gradient ∇L(v) of the regularized loss L is given by

∇L(v)/2 =

(cid:88)

k

S⊤X ⊤

k (XkSv − Yk)/n + λv =

S⊤MkSv −

(cid:88)

k

(cid:88)

k

S⊤X ⊤

k Yk/n + η

= Hv −

(cid:88)

k

S⊤X ⊤

k Yk/n,

where H := S⊤M S + λIm ∈ Rm×m, with M := M1 + M2 and Mk := X ⊤
R := H −1, we may write
1 Y1 + X ⊤

2 Y2)/n = RS⊤(M1w1 + M2w2) + RS⊤X ⊤

(cid:98)v = RS⊤(X ⊤

1 E1/n + RS⊤X ⊤

2 E2/n.

k Xk/n. Thus, setting

Now, one deduces the bias-variance decomposition
Etest( (cid:98)fRP ) = EDEx∼N (0,Σk)[( (cid:98)fRP (x) − x⊤w∗

where Vk := Vk,1 + Vk,2, with Vk,j :=

σ2
j
n

1)2] = EX1,E1,X2,E2∥S(cid:98)v − wk∥2
EX1,X2 tr S⊤MjSRS⊤ΣkSRS⊤,

Σk

= Bk + Vk,

Bk := EX1,X2∥SRS⊤(M1w1 + M2w2) − wk∥2
Σk

.

The variance terms Vk,j can be directly handled via FPT computations. We now look at the bias
term Bk. We first treat the case k = 1. One has

E∥SRS⊤(M1w1 + M2w2) − w1∥2
Σ
= E∥(SRS⊤(M1 + M2) − Id)w1 + SRS⊤M2δ∥2
Σ
= E∥(SRS⊤M − Id)w1∥2
Σ + E∥SRS⊤M2δ∥2
Σ
= E tr Γ(SRS⊤M − Id)Σ(M SRS⊤ − Id) + E tr ∆M2SRS⊤ΣSRS⊤M2
= tr ΓΣ + tr ΓSRS⊤M ΣM SRS⊤ − 2E tr ΓΣSRS⊤M + E tr ∆M2SRS⊤ΣSRS⊤M2
= tr ΓΣ + E tr ΣM SRS⊤ΓSRS⊤M − 2E tr ΓΣSRS⊤M
(cid:123)(cid:122)
(cid:125)
classical term (B)

+ E tr ∆M2SRS⊤ΣSRS⊤M2
,
(cid:125)
(cid:123)(cid:122)
extra term (ζ)

(cid:124)

(cid:124)

k Xk.
k ZkΣ1/2
k /(nλ) is an nk × d random matrix with iid entries from N (0.1/(nλ)). Thus,

where we recall that R := (S⊤M S + λIm)−1 and M := M1 + M2 with Mk = X ⊤
For the purposes of FPT computations, it might help to observe that Mk = λΣ1/2
Zk := XkΣ1/2
Mk = λM k,
M k = Σ1/2
M = λM ,
M = M 1 + M 2 = Σ1/2
R = R/λ,

1 + Σ1/2

k ZkΣ1/2
k ,

2 Z2Σ1/2

1 Z1Σ1/2

k Z ⊤

1 Z ⊤

2 Z ⊤

k Z ⊤

),

2

R = (S⊤M S + Im)−1 =

(cid:16)

S⊤Σ1/2

1 Z ⊤

1 Z1Σ1/2

1 S + S⊤Σ1/2

2 Z ⊤

2 Z2Σ1/2

2 S + Im

We need minimal linear pencils for the random matrices

k , where

(152)

(153)

(154)

(155)

(156)

(cid:17)−1

.

(157)

(158)

(159)

(160)

(161)

AM 1SRS⊤BSRS⊤,
AM SRS⊤BSRS⊤M
ASRS⊤M ,
AM 2SRS⊤BSRS⊤M 2,
, Σ1/2
2

1

in terms of the set of free variables {A, B, Σ1/2

, S, Z1, Z2, S⊤, Z ⊤

1 , Z ⊤

2 }. Observe that

tr AM SRS⊤BSRS⊤M
= tr AM1SRS⊤BSRS⊤M1 + tr AM2SRS⊤BSRS⊤M2 + 2 tr AM SRS⊤BSRS⊤M,
tr ASRS⊤M = tr ASRS⊤M1 + tr ASRS⊤M2.

34

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

Under review as a conference paper at ICLR 2025

For our business, it is therefore sufficient to only compute (minimal) linear pencils for

ASRS⊤M 1,
AM 1SRS⊤BSRS⊤,
AM 1SRS⊤BSRS⊤M 1,
AM 1SRS⊤BSRS⊤M 2,

(162)
(163)
(164)
(165)

where M k := Σ1/2

k Z ⊤

k ZkΣ1/2

k , R := (cid:0)S⊤M S + Im

(cid:1)−1

, M := M 1 + M 2.

Observe that without the S matrix (i.e taking m = d and S = Id), the four matrix expressions above
reduce to what we had in the classical case.

35

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

