We present a framework for automatically ordering image patches that enables in-depth analysis of dataset relationship to learnability of a classification task using convolutional neural network. An image patch is a group of pixels residing in a continuous area contained in the sample. Our preliminary experimental results show that an informed smart shuffling of patches at a sample level can expedite training by exposing important features at early stages of training. In addition, we conduct systematic experiments and provide evidence that CNN’s generalization capabilities do not correlate with human recognizable features present in training samples. We utilized the framework not only to show that spatial locality of features within samples do not correlate with generalization, but also to expedite convergence while achieving similar generalization performance. Using multiple network architectures and datasets, we show that ordering image regions using mutual information measure between adjacent patches, enables CNNs to converge in a third of the total steps required to train the same network without patch ordering. Adva nc e s in Deep Lear nin g (DL) and Conv olu tio na l Neura l Netw or ks (CNN ) have dram atic a l l y impro ve d the state-of-the -ar t in compu te r vision tasks. Many of these brea kth ro ug h s are attribute d to the succe ssiv e featu re extrac tion and an increa sin g abstr a ct repre se nta tion of the underly ing training dat a using multi-stag e simple oper ation s such as conv olutio n. These opera tion s posse ss seve ra l mod e l para m ete r s such as conv olution filter whic h are traine d to amplif y and refine infor m a tio n that are relev a n t to the classific a tio n, and to suppr e ss irrele v an t infor m atio n (Ian BID8 . The traini n g proce d u re uses backp ro p a ga tio n algorithm with super vision . This algorith m comb ine d with Stocha s t i c Gradie nt Desc e nt (SGD ), attem pts to minim iz e the over all erro r or devia tio n from true label by compu ti n g the error grad ien t of each para m e te r and by perfo rm in g small upda te s in the opposite directio n. Desp i t e their succ e ss, theore tic a l char ac te riz ation of deep learnin g and CNN s is still at its infanc y and valua b l e corre latio ns such as numbe r of layer s need ed to achie ve a certain perfo rm a n c e are not well under sto o d . However, the success of deep learning has spawned many research avenues in order to explain deep network's exceptional generalization performance BID19 BID14 BID16 Tishby and Zaslavsky, 2015) . One promising theoretical characterization of deep learning that supports an intuition that motivated this work is the characterization that uses an information theoretic view of feature extraction. In particular it is based on the information bottleneck (IB) method which is concerned with the problem of how one extracts an efficient representation of relevant information contained in a large set of features BID21 . BID19 proposes to study deep learning through the lens of information theory using the IB principle. In this characterization, deep learning is modeled as a representation learning. Each layer of a deep neural network can be seen as a set of summary statistics which contain some of the information present in the training set, while retaining as much information about the target output as possible BID19 . In this context a relevant information, of a cat vs dog classification task for instance, is the information pattern present in all the cat samples useful for predicting any picture of a cat. With this view, the amount of information relating the training set and the labels encoded in the hidden layers can be measured over the course of training (Tishby and Zaslavsky, 2015) . Inspired by this view, we use information theoretic measures of entropy extended to measure image characteristics, to develop preprocessing techniques that enable rob ust features extraction during training. One relev a nt insigh t prese nte d in these pape r s is that the goal of DL is to captu re and efficie n tly repr e se nt the relev a nt inform a tion in the input varia b le that desc rib e the outp u t variab le. This is equiv ale nt to the IB meth od whose goal is to find maxim a lly comp re sse d mappin g of the input while prese rvin g as much relev a nt inform ation of the output as possible . This chara cte riz a ti o n leads us to ask the questio n: In superv ise d learnin g, we are intere sted in good featu re repre se nta tio n s of the input patte rn that ena b l e good predictio n of the label BID9 . As a result, a training set for ima g e classific a tio n tasks that employ superv ise d learnin g, is constr uc te d with the help of huma n labele r . For instan ce , for a cat vs dog classifica tion proble m , the huma n labele r must cate go riz e each sample into eithe r one of the classe s. Durin g this proce ss, the labele r must recog niz e and classify each input usin g their own expe rie nc e and distin guis hing capa b ilitie s. Considering this, a natural question we first must answer before addressing the question above is: We proposed several automated patch ordering techniques to assess their impact on training and assess the relationship between dataset characteristics and training and generalization performances . Our methods rank, and reorder patches of every sample based on a standalone meas ure and based on similarit y between patches. We used traditional image similarity measures as well as information theory -based content measures of images to reconstruct training samples. We started off with theoretical foundations for measures used and highlighted the intuition regarding ordering and classification performance. We tested the proposed methods using several architectures, each effectively designed to achieve high accuracy on image classification tasks. The empirical evidence and our analysis using multiple datasets and Inception network architecture, suggest that training a convolutional neural network by supplying inputs that have some ordering, at patch level, according to some measure, are effective in allowing a gradient step to be taken in a direction that minimizes cost at every iteration. Specifically, our experiment s and CIFAR100 (right) datasets. Total training loss (top) and regularization loss (bottom) for Unmodified dataset, and datasets modified by applying Algorithm 1 using the MI metric and patch sizes 4x4, 8x8 and 16x16). The overall size of each sample is 32 by 32.show that supplying training sample such that the mutual information between adjacent patches is minimum, reduces the loss faster than all other techniques when optimizing a non-convex loss function. In addition, using these systematic approaches, we have shown that image characteristics and human recognizable features contained within training samples are uncorrelated with network performance. In other words, the view that CNNs learn combination of features in increasing abstraction does not explain their ability to fit images that have no recognizable features for the human eyes. Such a view also discounts the ability of the networks to fit random noise during training . Instead further investig a t i o n using theore tic a l chara cte riz a tio n s such as the IB metho d are nece ssa ry to form ally char a cte riz e learn ab il i t y of a given trainin g set using CNN .
Develop new techniques that rely on patch reordering to enable detailed analysis of data-set relationship to training and generalization performances.