The lack of crisp mathematical models that capture the structure of real-world
 data sets is a major obstacle to the detailed theoretical understanding of deep
 neural networks. Here, we first demonstrate the effect of structured data sets
 by experimentally comparing the dynamics and the performance of two-layer
 networks trained on two different data sets: (i) an unstructured synthetic data
set containing random i.i.d. inputs, and (ii) a simple canonical data set such
 as MNIST images. Our analysis reveals two phenomena related to the dynamics of
 the networks and their ability to generalise that only appear when training on
 structured data sets. Second, we introduce a generative model for data sets,
 where high-dimensional inputs lie on a lower-dimensional manifold and have
 labels that depend only on their position within this manifold. We call it the
 *hidden manifold model* and we experimentally demonstrate that training
 networks on data sets drawn from this model reproduces both the phenomena seen
 during training on MNIST. A major impediment for understanding the effectiveness of deep neural networks is our lack of mathematical models for the data sets on which neural networks are trained. This lack of tractable models prevents us from analysing the impact of data sets on the training of neural networks and their ability to generalise from examples, which remains an open problem both in statistical learning theory (Vapnik, 2013; Mohri et al., 2012) , and in analysing the average-case behaviour of algorithms in synthetic data models (Seung et al., 1992; Engel & Van den Broeck, 2001; Zdeborová & Krzakala, 2016) . Indeed, most theoretical results on neural networks do not model the structure of the training data, while some works build on a setup where inputs are drawn component-wise i.i.d. from some probability distribution, and labels are either random or given by some random, but fixed function of the inputs. Despite providing valuable insights, these approaches are by construction blind to key structural properties of real-world data sets. Here, we focus on two types of data structure that can both already be illustrated by considering the simple canonical problem of classifying the handwritten digits in the MNIST database using a neural network N ( LeCun & Cortes, 1998) . The input patterns are images with 28 × 28 pixels, so a priori we work in the high-dimensional R 784 . However, the inputs that may be interpreted as handwritten digits, and hence constitute the "world" of our problem, span but a lower-dimensional manifold within R 784 which is not easily defined. Its dimension can nevertheless be estimated to be around D ≈ 14 based on the neighbourhoods of inputs in the data set (Grassberger & Procaccia, 1983; Costa & Hero, 2004; Levina & Bickel, 2004; Facco et al., 2017; Spigler et al., 2019) . The intrinsic dimension being lower than the dimension of the input space is a property expected to be common to many real data sets used in machine leanring. We should not consider presenting N with an input that is outside of its world (or maybe we should train it to answer that the "input is outside of my world" in such cases). We will call inputs structured if they are concentrated on a lower-dimensional manifold and thus have a lower-dimensional latent representation. The second type of the structure concerns the function of the inputs that is to be learnt, which we will call the learning task. We will consider two models: the teacher task, where the label is obtained as a function of the high-dimensional input; and the latent task, where the label is a function of only the lower-dimensional latent representation of the input. structured inputs inputs that are concentrated on a fixed, lower-dimensional manifold in input space latent representation for a structured input, its coordinates in the lower-dimensional manifold task the function of the inputs to be learnt latent task for structured inputs, labels are given as a function of the latent representation only teacher task for all inputs, labels are obtained from a random, but fixed function of the high-dimensional input without explicit dependence on the latent representation, if it exists MNIST task discriminating odd from even digits in the MNIST database vanilla teacher-student setup Generative model due to Gardner & Derrida (1989) , where data sets consist of component-wise i.i.d. inputs with labels given by a fixed, but random neural network acting directly on the input hidden manifold model (HMF) Generative model introduced in Sec. 4 for data sets consisting of structured inputs (Eq. 6) with latent labels (Eq. 7) Table 1 : Several key concepts used/introduced in this paper. We begin this paper by comparing neural networks trained on two different problems: the MNIST task, where one aims to discriminate odd from even digits in the in the MNIST data set; and the vanilla teacher-student setup, where inputs are drawn as vectors with i.i.d. component from the Gaussian distribution and labels are given by a random, but fixed, neural network acting on the high-dimensional inputs. This model is an example of a teacher task on unstructured inputs. It was introduced by Gardner & Derrida (1989) and has played a major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective, particularly within the framework of statistical mechanics (Seung et al., 1992; Watkin et al., 1993; Engel & Van den Broeck, 2001; Zdeborová & Krzakala, 2016; Advani & Saxe, 2017; Aubin et al., 2018; Barbier et al., 2019; Goldt et al., 2019; Yoshida et al., 2019) , and also in recent statistical learning theory works, e.g. (Ge et al., 2017; Li & Y., 2017; Mei & Montanari, 2019; Arora et al., 2019) . We choose the MNIST data set because it is the simplest widely used example of a structured data set on which neural networks show significantly different behaviour than when trained on synthetic data of the vanilla teacher-student setup. Our reasoning then proceeds in two main steps: 1. We experimentally identify two key differences between networks trained in the vanilla teacherstudent setup and networks trained on the MNIST task (Sec. 3). i) Two identical networks trained on the same MNIST task, but starting from different initial conditions, will achieve the same test error on MNIST images, but they learn globally different functions. Their outputs coincide in those regions of input space where MNIST images tend to lie -the "world" of the problem, but differ significantly when tested on Gaussian inputs. In contrast, two networks trained on the teacher task learn the same functions globally to within a small error. ii) In the vanilla teacher-student setup, the test error of a network is stationary during long periods of training before a sudden drop-off. These plateaus are well-known features of this setup (Saad & Solla, 1995; Engel & Van den Broeck, 2001 ), but are not observed when training on the MNIST task nor on other datasets used commonly in machine learning.
We demonstrate how structure in data sets impacts neural networks and introduce a generative model for synthetic data sets that reproduces this impact. The paper studies how different settings of data structure affect learning of neural networks and how to mimic behavior on real datasets when learning on a synthetic one.