Large Deviations for Accelerating Neural Networks Training
Artificial neural networks (ANNs) require tremendous amount of data to train
on. However, in classification models, most data features are often similar
which can lead to increase in training time without significant improvement in
the performance. Thus, we hypothesize that there could be a more efficient way
to train an ANN using a better representative sample. For this, we propose the
LAD Improved Iterative Training (LIIT), a novel training approach for ANN using
large deviations principle to generate and iteratively update training samples
in a fast and efficient setting. This is exploratory work with extensive
opportunities for future work. The thesis presents this ongoing research work
with the following contributions from this study: (1) We propose a novel ANN
training method, LIIT, based on the large deviations theory where additional
dimensionality reduction is not needed to study high dimensional data. (2) The
LIIT approach uses a Modified Training Sample (MTS) that is generated and
iteratively updated using a LAD anomaly score based sampling strategy. (3) The
MTS sample is designed to be well representative of the training data by
including most anomalous of the observations in each class. This ensures
distinct patterns and features are learnt with smaller samples. (4) We study
the classification performance of the LIIT trained ANNs with traditional batch
trained counterparts.
Sreelekha Guggilam, Varun Chandola, Abani Patra
# Large Deviations for Accelerating Neural Networks Training

###### Abstract

Artificial neural networks (ANNs) require tremendous amount of data to train on. However, in classification models, most data features are often similar which can lead to increase in training time without significant improvement in the performance. Thus, we hypothesize that there could be a more efficient way to train an ANN using a better representative sample. For this, we propose the LAD Improved Iterative Training (LIIT), a novel training approach for ANN using large deviations principle to generate and iteratively update training samples in a fast and efficient setting. This is exploratory work with extensive opportunities for future work. The thesis presents this ongoing research work with the following contributions from this study: (1) We propose a novel ANN training method, LIIT, based on the large deviations theory where additional dimensionality reduction is not needed to study high dimensional data. (2) The LIIT approach uses a Modified Training Sample (MTS) that is generated and iteratively updated using a LAD anomaly score based sampling strategy. (3) The MTS sample is designed to be well representative of the training data by including most anomalous of the observations in each class. This ensures distinct patterns and features are learnt with smaller samples. (4) We study the classification performance of the LIIT trained ANNs with traditional batch trained counterparts.

Large deviations, anomaly detection, high-dimensional data, multivariate time series 00 Month
2. We present four LAD score based sampling strategies to design the MTS. Obtaining the LAD score based on a large deviations principle is computationally inexpensive. Therefore, one can analyze large and high dimensional datasets without additional dimensionality reduction procedures allowing more accurate and cost effective scoring schema.
3. The use of MTS which is a smaller training sample reduces the cost of computational time significantly for large datasets.
4. We perform an empirical study on publicly available classification benchmark datasets to analyze the performance of the proposed method.

The work presented here is limited to simple classification based neural networks. Future work will include extending it to more complex ANNs.

## 2 Related Work

In this section, we provide a brief overview of sensitivity to training samples and speed of neural network training.

Artificial neural networks are powerful for general classification. However, its excellent performance often depends largely on a huge training set. A large body of research exists that study the impact of training data size on neural network learning [6, 2]. In particular, it is evident that smaller training data leads to less efficient models. However, the vast computational expense associated with training on large sets of data makes the need to improve training practices essential, specially for online or real-time models.

Many methods that try to model faster neural networks exist. For instance, Wang et al. [7] use batch normalization in deep neural networks to improve the convergence rates. Zhong et al. [8] work on image classification using their agile convolution neural network SatCNN, for quick and effective learning with small convolutional kernels and deep convolutional layers. However, these works are limited to the domain problems and cannot be easily scaled to other data types.

Another alternative to improve the training speed can be by modifying the training samples. For instance, studies like Shanker et al. [5] look at the effect of standardization of data on the learning of the neural network. Kavzoglu [3] emphasizes on characteristics of training samples and uses representative training to improve the classification. These methods, however, fail to study the impact of smaller data on model performance and efficiency.

In this part of the thesis, we propose a novel training strategy that can be generalized across domains. The method is used to replicate the true representation of the training features in a smaller sample which can be in turn used for faster training and convergence. Due to the proper representation of even the most extreme observations, this method ensures faster learning with competitive performance.

## 3 Methodology

The most important aspect of classification models is the adequacy of the representative training samples for each class. Although the size of the training data is of considerable importance, acquiring a large number of representative training data may be impractical where a large number of classes are involved. In particular, since most observations within each true class have similar features, multiple samples add low value in terms of novel information/pattern. In this section, we describe the traditional batch training approach in brief followed by the LAD Improved Iterative Training approach. We present 4 sampling strategies used in the LIIT training and their respective algorithms.

### Definitions and Terminology

Before describing the detailed methodology, we list out the terminology and corresponding definitions that are used for this study.

**Definition 1**.: **LAD Score** is the Large deviations Anomaly Detection (LAD) generated anomaly score for each observation in the data.
**Definition 2**.: **Full-Training Data** is the available complete training dataset for the ANN. It must be noted that only a subset of the Full Training Data might be used to train the ANN in the LIIT approach. Hence we present a different terminology to differentiate it from the training data.

**Definition 3**.: **Batch Training** is the traditional ANN training method using mini-batches of training data.

**Definition 4**.: **Modified Training Sample (MTS)** is a smaller sample generated from the training data using a specific sampling algorithm.

**Definition 5**.: **LAD Improved Iterative Training (LIIT)** is the novel improved batch training approach to train the ANN.

### Classification Neural Network

For this analysis, we look at a basic classification algorithm. Figure 1 shows the architecture of a simple three layer dense neural network.

The model is trained using full training samples with the convergence criterion set to zero validation loss for 5 epochs with the maximum number of epochs set to 180. Three different activation functions, RELU, Tanh, Softmax are used for the three consecutive dense layers respectively. A simple model was chosen to study the proof of concept of the representative sampling strategy presented in the part of the thesis. Further studies are needed to understand the relation between the model choice and training sampling techniques.

### LAD Improved Iterative Training of The Neural Network

Traditionally, in batch training, the full training data is divided into smaller samples or batches. The ANN learns from each batch sequentially till all the observations from the full training data are exhausted, as demonstrated in Figure 2. In the LIIT training, we iteratively design and update the modified training samples, MTS, from the full training data. At each iteration, we train the ANN using batch training on MTS till convergence. This partially trained model is then tested on the full training data to identify potential learning flaws. Since the current work is limited to classification models, the learning flaws include the misclassified data. The misclassified data is then used to derive the updated MTS which is used to retrain the ANN. The process is illustrated in Figure 3. This is inspired by Boosting techniques [4] where the subset creation depends on the previous model. However, unlike in boosting setting, we retrain the same ANN. 1

Figure 1: Simple Classification Neural Network: The figure illustrates a dense neural network to classify data into 3 classes. The network takes an input of 10 dimensions and returns scores used to assign each class.
To determine and extract the MTS sample, any sampling algorithm can be used. However, to ensure a good representation, we designed four LAD score based sampling algorithms along with the random sampling approach which is used as a baseline. The following are the sampling strategies used in our analysis:

1. **LAD Anomaly only (Repeated Entry)**: Observations with the highest anomaly scores in each true class are added to the training batch. Multiple copies of the observation can be added over iterations when the model fails to classify them after numerous re-training. See Algorithm 1.
2. **LAD Anomaly + Normal (Unique Entry)**: Equal parts of the high and low anomaly score observations are sampled for each true class. The final training batch contains a unique set of observations with no duplicate entries. See Algorithm 2.
3. **LAD Anomaly only (Unique Entry)**: This is similar to the **LAD Anomaly only (Repeated Entry)** approach. Observations with the highest anomaly scores in each true class are added to the training batch. However, the final training batch contains a unique set of observations with no duplicate entries. See Algorithm 3.
4. **LAD Quantile Samples (Repeated Entry)**: The observations are sampled using different quantiles of the anomaly score for each true class. Multiple copies are maintained in the training batch to ensure weighting from under-represented latent classes within each known true class. See Algorithm 4.
5. **Random**: In this model, we use random sampling from the available data. See Algorithm 5.

For this part, we sample \(\sim 5-6\%\) of the full training data at each iteration that is later added to the modified training sample. We ensure equal weights for all true classes for the analysis. The LIIT approach is implemented with 6 iterations (1 initial and 5 updates) which brings to \(\sim 30\%\) of the full training sample used in the LIIT approach.

## 4 Experiments

In this section, we evaluate the classification performance of the simple neural networks on real data when trained using LAD sub-sampled data. We focus on the performance of the neural networks under different training and sampling settings.

The following experiments have been conducted to study the model:

Figure 2: Mini-Batch Training Algorithm
1. Computational Expense: The LIIT trained ANN model's ability to train on a smaller set of training samples and converge faster is compared to the fully trained model.
2. Classification Performance: The overall performance of the sub-sampled models on multiple benchmark datasets is studied. For this analysis, we consider Area Under the Curve (AUC) as the performance metric to study classification.
3. Stability to Perturbations: Perturbations upto 8% are added to the test data which is used to study the change in performance in all models.

To maintain fair comparison, the number of epochs is fixed to a maximum count of 180 for the ANN model trained on the full training data a.k.a. the full model and 30 per iteration of all the LIIT trained ANNs (totaling to 180 epochs for complete training). For each trained ANN, we evaluate performance on 5 independent reruns. The average results are presented for all evaluations.

Figure 3: LIIT Training Algorithm
```
0: Dataset \(X\) of size \((n,d)\), number of iterations \(N_{iter}\), threshold \(th\), number of true classes in the data \(K\), sample size from each class \(c_{size}\), number of iterations \(i_{iter}\), ANN classification model \(model_{iiti}\). Initialization: Split data into \(x_{train},x_{test},x_{val},y_{train},y_{test},y_{val}\) (train, test and validation) Derive LAD score \(ana_{score}\) for all observations in training data i.e. \(ana_{score}=LAD(x_{train},y_{train})\)
1:\(MTS=[]\) (create empty MTS sample indices list)
2:for each class \(k\)do
3: Generate list of indices of all observations in class \(k\), \(ind_{k}\)
4: Subset anomaly scores for each class \[ana_{score_{k}}=ana_{score}[ind_{k}]\]
5: Identify top \(c_{size}\) observations with least anomaly scores and add them samples to the \(MTS\) sample i.e. (most non-anomalous observations)
6:for each iteration \(i\leq i_{iter}\)do
7: Fit the ANN on \(MTS\) using batch training, \[model_{iiti}.fit(x_{train}[MTS],y_{train}[MTS])\]
8: Predict model classification on \(x_{train}\), \(z_{pred}=model_{iiti}.predict(x_{train})\)
9: Identify all miss-classified observations' indices in training data \[err_{inds}=np.vhere(z_{pred}.!=y_{train})\]
10:for each class \(k\)do
11: Identify all miss-classified observations \(ind_{err_{k}}\)
12: Subset anomaly scores for miss-classified data in class \(k\) \[ana_{err_{k}}=ana_{score}[ind_{err_{k}}]\]
13: Identify \(c_{size}\) observations with highest anomaly scores from \(ind_{err_{k}}\) i.e. (most anomalous observations) and add them to \(MTS\) sample.
14:endfor
15:endfor
16:endfor
```

**Algorithm 1** LAD Anomaly only (Repeated Entry)

### 4.1 Datasets

We consider a variety of publicly available benchmark data sets from the UCI-ML repository [1]) (See Table 1 ) for the experimental evaluation. For training, test and validation, the data was randomly split into 80%, 10% and 10% of the data respectively.

#### Computational Time

In this section, we look at the time taken by each ANN to train on the datasets. Since the LIIT trained ANNs use only one-third of the full training data, the training time is evidently lower as compared to training for the full model. This can be clearly seen in the Figures 4.
```
0: Dataset \(X\) of size \((n,d)\), number of iterations \(N_{iter}\), threshold \(th\), number of true classes in the data \(K\), sample size from each class \(c_{size}\), number of iterations \(i_{iter}\), ANN classification model \(model_{limit}\). Initialization: Split data into \(x_{train},x_{test},x_{val},y_{train},y_{test},y_{val}\) (train, test and validation) Derive LAD score \(ana_{score}\) for all observations in training data i.e. \(ana_{score}=LAD(x_{train},y_{train})\)
1:\(MTS=[]\) (create empty MTS sample indices list)
2:for each class \(k\)do
3: Generate list of indices of all observations in class \(k\), \(ind_{k}\)
4: Subset anomaly scores for each class \[ana_{score_{k}}=ana_{score}[ind_{k}]\]
5: Identify top \(c_{size}\) observations with least anomaly scores and add them samples to the \(MTS\) sample i.e. (most non-anomalous observations)
6:for each iteration \(i\leq i_{iter}\)do
7: Fit the ANN on \(MTS\) using batch training, \[model_{lift}.fit(x_{train}[MTS],y_{train}[MTS])\]
8: Predict model classification on \(x_{train}\), \(z_{pred}=model_{limit}.predict(x_{train})\)
9: Identify all miss-classified observations' indices in training data \[err_{inds}=np.where(z_{pred}!=y_{train})\]
10:for each class \(k\)do
11: Identify all miss-classified observations \(ind_{err_{k}}\)
12: Subset anomaly scores for miss-classified data in class \(k\) \[ana_{err_{k}}=ana_{score}[ind_{err_{k}}]\]
13: Identify \(c_{size}/2\) observations each for the lowest and highest anomaly scores from \(ind_{err_{k}}\) i.e. (most anomalous as well as least anomalous observations) and add them to the \(MTS\) sample indices.
14:endfor
15: Remove repeated indices in the updated modified training sample, \[MTS=unique(MTS)\]
16:endfor
17:endfor
```

**Algorithm 2** LAD Anomaly + Normal (Unique Entry)
```
0: Dataset \(X\) of size \((n,d)\), number of iterations \(N_{iter}\), threshold \(th\), number of true classes in the data \(K\), sample size from each class \(c_{size}\), number of iterations \(i_{iter}\), ANN classification model \(model_{limit}\). Initialization: Split data into \(x_{train},x_{test},x_{val},y_{train},y_{test},y_{val}\) (train, test and validation) Derive LAD score \(ana_{score}\) for all observations in training data i.e. \(ana_{score}=LAD(x_{train},y_{train})\)
1:\(MTS=[]\) (create empty MTS sample indices list)
2:for each class \(k\)do
3: Generate list of indices of all observations in class \(k\), \(ind_{k}\)
4: Subset anomaly scores for each class \[ana_{score_{k}}=ana_{score}[ind_{k}]\]
5: Identify top \(c_{size}\) observations with least anomaly scores and add them samples to the \(MTS\) sample i.e. (most non-anomalous observations)
6:for each iteration \(i\leq i_{iter}\)do
7: Fit the ANN on \(MTS\) using batch training, \[model_{lift}.fit(x_{train}[MTS],y_{train}[MTS])\]
8: Predict model classification on \(x_{train}\), \(z_{pred}=model_{limit}.predict(x_{train})\)
9: Identify all miss-classified observations' indices in training data \[err_{inds}=np.where(z_{pred}!=y_{train})\]
10:for each class \(k\)do
11: Identify all miss-classified observations \(ind_{err_{k}}\)
12: Subset anomaly scores for miss-classified data in class \(k\) \[ana_{err_{k}}=ana_{score}[ind_{err_{k}}1]\]
13: Identify \(c_{size}\) observations with highest anomaly scores from \(ind_{err_{k}}\) i.e. (most anomalous observations) and add them to \(MTS\) sample.
14:endfor
15: Remove repeated indices in the updated modified training sample, \[MTS=unique(MTS)\]
16:endfor
17:endfor
```

**Algorithm 3** LAD Anomaly only (Unique Entry)
of the models. It is discernible that the Quantile Sampling along with LIIT trained ANN model is on par with the fully trained model.

#### 4.1.3 Stability to Perturbations

Since the training samples have a significant influence on the model's learning and performance, we try to look at the stability of the model to various perturbations in the test data. For this, random noise is sampled from a multivariate normal distribution with the \(0-8\%\) of the training data mean and variance and is added to all the observations in the test data. Each ANN's performance is evaluated in these settings for all benchmark datasets. The final classification performances are seen in Figures 5. It was interesting to note that different datasets had better and relatively more stable performances using different sampling strategies.

Now, to see the individual changes in performance to perturbations, we look at the raw change in AUC values due to the addition of perturbations for all models. Figures 6 show the change in performance for different datasets. In particular, Figures 6 a and 6 b show a group of datasets that show better performance using Quantile (Repeated), while Figures 6 c - 6 e show performance on datasets where Anomaly (Unique), Anomaly + Normal (Repeated) and Anomaly (Repeated) sampling approaches have respectively outperformed.

It can be seen that the Quantile Sample Trained Model has a higher mean AUC as well as lower deviation in AUC than the fully trained model in most datasets.
Here, we can see that different LIIT models outperform for different datasets. We hypothesize that the data distribution and heterogeneity play important role in the overall performance and stability. We intend to continue the study of the proposed hypothesis as future research.

## 5 Conclusion

We present a new training strategy for enhancing the learning speed of a neural network whilst maintaining the performance of the model. We present the LAD Improved Iterative Training (LIIT) which is an improved iterative training version of the traditional batch training approach. The LIIT approach uses a modified training sample (MTS) generated and updated using a LAD score based sampling approach that ensures enough representation of extreme and rare behaviours. In particular, the LAD score based Quantile Sampling approach allows ample heterogeneity within the sample data. We study the classification performance of the LIIT trained ANN in comparison with ANN trained on full training data on real benchmark datasets. Though the current research is limited to simple classification neural networks, the work has immense research potential. The LIIT training approach combined with specific LAD sampling methodology might draw out the best performance in a dataset based on the data characteristics. Future studies might help understand the impact of data heterogeneity and sampling method on the performance of ANN.
\begin{table}
\begin{tabular}{|l|r|r|r|} \hline Name & \(N\) & \(d\) & \(c\) \\ \hline Ecoli & 336 & 7 & 8 \\ Imgseg & 2310 & 18 & 7 \\ Skin & 245057 & 4 & 2 \\ Shuttle & 58000 & 10 & 2 \\ Wisc & 699 & 9 & 2 \\ Iono & 351 & 33 & 2 \\ Zoo & 101 & 16 & 7 \\ Letter & 20000 & 16 & 26 \\ Comm And Crime & 1994 & 102 & 2 \\ Vowel & 990 & 10 & 11 \\ Fault & 1941 & 28 & 2 \\ Sonar & 208 & 60 & 2 \\ Balance-Scale & 625 & 4 & 3 \\ Pageb & 5473 & 11 & 2 \\ Spambase & 4601 & 58 & 2 \\ Wave & 5000 & 22 & 2 \\ Tae & 151 & 3 & 3 \\ Thy & 215 & 5 & 3 \\ Opt Digits & 5620 & 63 & 2 \\ Concrete & 1030 & 9 & 2 \\ \hline \end{tabular}
\end{table}
Table 1: classification Benchmark Datasets: Description of the benchmark data sets used for evaluation of the classification detection capabilities of the proposed model. \(N\) - number of instances, \(d\) - number of attributes, \(c\) - number of true classes in the data set.
## References

* [1]D. Dheeru and E. Karra Taniskidou (2017) UCI machine learning repository. External Links: Link Cited by: SS1.
* [2]J. Djolonga, J. J. Yung, M. Tschannen, R. Romijnders, L. Beyer, A. Kolesnikov, J. Puigcerver, M. Minderer, A. D'Amour, D. Moldovan, et al. (2021) On robustness and transferability of convolutional neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition16458-16468. Cited by: SS1.
* [3]K. Kavzoglu (2009) Increasing the accuracy of neural network classification using refined training data. Environmental Modelling & Software24 (7), pp. 850-858. Cited by: SS1.
* [4]R. E. Schapire (2003) The boosting approach to machine learning: an overview. Nonlinear estimation and classification149-171. Cited by: SS1.
* [5]M. Shanker, M. Y. Hu, and M. S. Hung (1996) Effect of data standardization on neural network training. Omega24 (4), pp. 385-397. Cited by: SS1.
* [6]D. Soekhoe, D. Van Der Putten, and A. Plaat (2016) On the impact of data set size in transfer learning using deep neural networks. International symposium on intelligent data analysis, pp. 50-60. Cited by: SS1.
* [7]J. Wang, S. Li, Z. An, X. Jiang, W. Qian, and S. Ji (2019) Batch-normalized deep neural networks for achieving fast intelligent fault diagnosis of machines. Neurocomputing329, pp. 53-65. Cited by: SS1.
* [8]Y. Zhong, F. Fei, Y. Liu, B. Zhao, H. Jiao, and L. Zhang (2017) Satcnn: satellite image dataset classification using agile convolutional neural networks. Remote sensing letters8 (2), pp. 136-145. Cited by: SS1.

Figure 4: Computation time for different datasets: The figures illustrate the computation time for different LIIT trained ANN models in comparison to the ANN trained on full training data (Full model).




