In this paper, we empirically investigate the training journey of deep neural networks relative to fully trained shallow machine learning models. We observe that the deep neural networks (DNNs) train by learning to correctly classify shallow-learnable examples in the early epochs before learning the harder examples. We build on this observation this to suggest a way for partitioning the dataset into hard and easy subsets that can be used for improving the overall training process. Incidentally, we also found evidence of a subset of intriguing examples across all the datasets we considered, that were shallow learnable but not deep-learnable. In order to aid reproducibility, we also duly release our code for this work at https://github.com/karttikeya/Shallow_to_Deep/ Analyzing the temporal journey taken by deep neural networks (DNNs) during training has elicited a lot of attention recently. The authors in BID0 suggested that DNNs learn simple patterns first, before memorizing. More specifically, they posit that real world datasets are littered with easy examples characterized by simple patterns that are learned in the initial epoch(s) before the conquest of hard examples in the training dataset. Tishby et al BID25 conjectured that DNN training was characterized by two distinct phases consisting of an initial fitting phase (memorization) and a subsequent compression phase. While this claim was questioned in BID24 , the authors do remark that when an input domain consists of a subset of task-relevant and task-irrelevant information, hidden representations do compress the task-irrelevant information. These works do suggest that the easy-vs-hard dichotomy in real-world datasets does influence the learn- ing in DNNs and goad a data-dependent approach towards understanding the capacity of DNNs. Taking cue from this, we strive to contribute to this growing body of literature by bringing in another viewpoint: The dichotomy between shallow learnable examples and deep learnable examples in the dataset. More specifically we try to address the questions:1. Is the notion of easiness same for models with as different parameterizations and architectures as shallow machine learning models and deep networks the same? And hence is attached to the example independently of a model?2 . If we are to investigate the examples that a DNN learns to correctly classify over the training batches, do we observe a shallow learnable to deep learnable regime change?3 . Are there examples that are shallow learnable but somehow a DNN with a far better overall accuracy fails to classify? At the heart of this quest is to understand if shallow learnability is a good proxy for the easiness of an example.We'd like to reiterate that the motivation behind this work is to obtain insights into the changing scenery of the conquest of the training dataset experienced by deep neural networks and not to delineate the nature of compositional functions that DNNs can learn and shallow algorithms cannot or comment on the amount of training data required to do so. In BID19 , the authors have already shown how DNNs can approximate the class of compositional functions as well as shallow networks but with exponentially lower number of training parameters and sample complexity.The rest of the paper is organized as follows. In section 2, we present the quantitative methodology we used to answer these questions raised above. In section 3, we showcase our empirical experiments with the results covered in section 4. We conclude the paper in section 5. In this work, we track the training of DNNs relative to shallow machine learning models. We showcase some results on analyzing the training trajectory of the DNNs relative to SVM and RF on three different datasets. Empirically, we observe that the during training the Deep Network quickly learns shallow classifiable easy examples first and then learns the hard examples in the later epochs. Furthermore, we find that the notion of hardness of an example is largely independent of the model being used and can be evaluated reliably using a shallow learning model. This observation allows for a procedural slicing of the training set into easy and hard categories that can improve network training. We also report a slightly surprising finding pertaining to the existence of a subset of examples in all the datasets considered that were shallow-classifiable but not deep-classifiable.
We analyze the training process for Deep Networks and show that they start from rapidly learning shallow classifiable examples and slowly generalize to harder data points.