The lack of crisp mathematical models that capture the structure of real-world
 data sets is a major obstacle to the detailed theoretical understanding of deep
 neural networks. Here, we first demonstrate the effect of structured data sets
 by experimentally comparing the dynamics and the performance of two-layer
 networks trained on two different data sets: (i) an unstructured synthetic data
set containing random i.i.d. inputs, and (ii) a simple canonical data set such
 as MNIST images. Our analysis reveals two phenomena related to the dynamics of
 the networks and their ability to generalise that only appear when training on
 structured data sets. Second, we introduce a generative model for data sets,
 where high-dimensional inputs lie on a lower-dimensional manifold and have
 labels that depend only on their position within this manifold. We call it the
 *hidden manifold model* and we experimentally demonstrate that training
 networks on data sets drawn from this model reproduces both the phenomena seen
 during training on MNIST.
We demonstrate how structure in data sets impacts neural networks and introduce a generative model for synthetic data sets that reproduces this impact. The paper studies how different settings of data structure affect learning of neural networks and how to mimic behavior on real datasets when learning on a synthetic one.