BGADAM: Boosting based Genetic-Evolutionary ADAM for Neural Network Optimization
Jiyang Bai Yuxiang Ren Jiawei Zhang
cs.NE cs.LG


 BSUV-Net: A Fully-Convolutional Neural Network for
 Background Subtraction of Unseen Videos
    
  M. Ozan Tezcan, Prakash Ishwar, Janusz Konrad
    This work was supported in part by ARPA-E under agreement DE-AR0000944 and by the donation of Titan GPUs from NVIDIA Corp.


  Department of Electrical and Computer Engineering

  Boston University, Boston, MA 

   


    
==================================================================================================================================================================================================================================================================================




For various optimization methods, gradient descent-based algorithms can achieve outstanding performance and have been widely used in various tasks. Among those commonly used algorithms,  owns many advantages such as fast convergence with both the momentum term and the adaptive learning rate. However, since the loss functions of most deep neural networks are non-convex,  also shares the drawback of getting stuck in local optima easily. To resolve such a problem, the idea of combining  with base learners is introduced to rediscover the best solutions. Nonetheless, from our analysis, the idea of combining  with a batch of base learners still has its shortcomings. The effectiveness of  can hardly be guaranteed if the unit models converge to close or the same solutions. To resolve this problem and further maximize the advantages of  with base learners, we propose to implement the  strategy for input model training, which can subsequently improve the effectiveness of . In this paper, we introduce a novel optimization algorithm, namely Boosting based Genetic  (). With both theoretic analysis and empirical experiments, we will show that adding the  strategy into the  model can help models jump out the local optima and converge to better solutions.









§ INTRODUCTION


Deep learning models have achieved impressive success in many areas, which include the computer vision <cit.>, natural language processing <cit.> and recently appeared graph neural networks <cit.>. By owning a large number of variables and non-linear functions (e.g., the Relu function <cit.>), deep learning models can fit extremely complex data and learn the hidden representations of them. To let the deep learning models achieve outstanding performance, one of the most important procedures is training the models with appropriate optimization algorithms. During the training process of deep learning models, the essence is to search for the global optima of the loss functions. This process can be represented by

    min_w∈𝒲 f(X,y; w)

Here, f(· ,· ;w) denotes the loss function with variables w∈𝒲, where 𝒲 is the solution space of variables; X and y denote the features and labels of the training data respectively. Up to now, the most commonly used optimization algorithms are based on the gradient descent, for example the stochastic gradient descent (SGD), SGD with momentum <cit.>,  <cit.>,  <cit.> and  <cit.>. Among these algorithms,  is the most widely used one and has been proven to be powerful in various tasks thanks to its fast convergence rate. However,  still has the common drawback of easily converging to the local optima and getting stuck into it, especially when dealing with the non-convex problems <cit.>. To solve the problem of converging to the local optima, the idea of integrating  and multiple learners have been proposed such as  <cit.>, which can combine the advantages of both the  and the . During the training process, initially multiple input models are trained simultaneously. From these input models, a batch of (parent) model pairs are selected, and  is implemented to the variables of these parent models after they converge to the local optima. Applying the  can help the trained parent models “jump out” of local optimal points when dealing with the non-convex problem. The “jump out” operation is implemented through reorganizing the variables of the parent models. Even though  with multiple learners has the potential capability of approaching more optimized solutions, the actual effect still can not be guaranteed. If most of the input models, or parent models converge to the close or even the same local optima, the  operation might become meaningless since the output models of the variables reorganizing still locate at the local optima. We denote this phenomenon of failing to jump out of local optima as the “local sticking”. To avoid the “local sticking” problems and increase the capability of converging to better solutions, one concrete idea is to let the parent models converge to divergent local optima. In this way, the  is more likely to generate models with variables at different locations and subsequently increase the possibility of converging to better solutions and achieving better performance.























In this paper, we will propose a new optimization algorithm, namely Boosting based Genetic ADAM () to solve the “local sticking” problem mentioned above and further achieve better performance compared with  algorithm. With the support of the boosting strategy,  can improve the performance of the optimization by differentiating the parent models, and meanwhile guarantee the convergence of the algorithm theoretically. Boosting strategy <cit.> utilizes the interactions among base learners to accomplish a more effective model training. The interaction is achieved by redistributing the dataset to modify the training set for each input model. In each training iteration, the  strategy samples training data from the training dataset with replacement subject to different weights. After the training process of one input model, the weights of all the data samples will be updated according to the classification results of this model. Based on the updated weights, the training set will be sampled under the updated distribution, and newly sampled training data will be assigned to the next input model. Meanwhile, this newly sampled dataset inclines to contain more data misclassified by the previous input model. In this way, by proposing the boosting strategy, different parent models will be trained with different training examples. In other words, the parent models trained by different training set will subsequently converge to divergent local optima with a higher probability. 
Therefore,  can effectively diversify the learned parent models, which can further improve the learning performance.

The following part of the paper is organized as follows. In Section <ref>, we will talk about some related works. In Section <ref>, we will cover more details about our proposed  algorithm, whose effectiveness will be analyzed in Section <ref>, and performance will be tested with extensive experiments in Section <ref>. Finally, we will conclude this paper in Section <ref>.








§ RELATED WORKS
 

Deep Learning Models: Deep learning models have achieved state-of-the-art results in recent years, whose representative examples include feed-forward deep neural networks (DNN) <cit.>,  convolutional neural networks (CNN) <cit.> <cit.>, and recurrent neural networks (RNN) <cit.>. DNN models mainly consist of several layers of the linear transforms and the non-linear activation functions. The combination of these linear and non-linear transforms between input features and outputs enables the DNN to fit more complex patterns hidden in the training data. Inspired by the great success of DNN models, more attention had been focused on transplanting the deep model ideas onto other types of data such as the image type and the sequential type data. To deal with the image data, CNN has been proposed and already shown outstanding performance on various computer vision tasks; for the sequential data, the RNN model came up. RNN <cit.> models are connectionist models with the ability to pass information across sequence elements while conducting sequential data one element at a time. Besides, there also exist many other types of deep learning models to tackle different types of data, e.g., the graph neural networks <cit.>, graph convolutional networks <cit.> and graph attention networks <cit.> to deal with graph type data; the Generative Adversarial Networks (GAN) <cit.> to train discriminative models and generative models, etc.





Genetic Algorithm: Genetic algorithm <cit.> is a family of computational algorithms inspired by creature evolution in the natural environment. It encodes a potential solution to a specific problem on chromosome-like data structures and applies recombination (reorganization) operators to these structures <cit.>. This process is also called crossover <cit.>. Besides,  also adds random mutations on the solutions to mimic the natural world's real gene mutation process. In this way, for some models stuck in local optimal points,  can help them jump out by switching specific variables. During the generating process, the above crossover and mutation operations are implemented to a bunch of models, which are called the population set. After that, the models with the worst performance (the fitness score) will be eliminated. Finally, individuals with relatively better performance are retained.


Optimization Algorithms: The main purpose of the optimization algorithms is to closely approach the best solution of the loss functions during the backpropagation. So far most commonly used optimizers are based on the gradient descent <cit.>, such as the stochastic gradient descent (SGD), the  <cit.>, the  <cit.> and the  <cit.>. 
SGD performs a variable update for each training example X[i,:] and label y[i], it can be expressed as:

    w = w - η·∇_w f(X[i,:], y[i])

where η is the learning rate and ∇_w is the derivative of the loss function regarding variable w. The advantages of SGD include fast speed and getting rid of redundancy <cit.>. SGD with momentum <cit.> is a method that helps accelerate SGD in the relevant directions. 
SGD with momentum updates variables with the following equations:

    v_t =  γ·v_t-1 + η·∇_w f(w)
       w_t = w_t -v_t

where γ is the momentum term weight. The momentum term accelerates variable updates for dimensions whose gradients are in the same direction as historical gradients and decelerates updates for dimensions whose gradients are the reverse. Therefore its convergence process will be faster. However, the unified learning rate for all variables can lead to some problems. If variables have different scales, we should not update them with the same rate. Some variant algorithms have been proposed to solve this problem, such as the  and the .  <cit.> adapts the learning rate to the parameters, performing larger updates for infrequent and smaller updates for frequent parameters.  <cit.> is a refined version of the  that applies the moving average of the historical gradients to adapt the learning rates.  <cit.> is proposed based on the SGD and the momentum, which computes individual adaptive learning rates for different variables. Similar to the SGD with momentum and the ,  records the first-order momentum m_t and the second-order momentum v_t of the gradients and computes the bias-corrected version of them respectively. 


Ensemble Learning: Ensemble learning <cit.> tries to train multiple learners to solve the same problem. In contrast to ordinary learning approaches that attempt to construct one model from the training data, ensemble methods try to construct a set of learners and combine them. The commonly used ensemble methods include boosting and bagging. Boosting <cit.> generally creates a batch of weak learners, and each learner is trained based on the sampled training data under the distribution decided by the previous learner. In other words, each learner can focus more on the data samples being misclassified by the previous learner. Bagging adopts the bootstrap distribution for generating different base learners. It applies bootstrap sampling <cit.> to obtain the data subsets for training the base learners.









§  LEARNING ARCHITECTURE
 


In this section, we will introduce more details about the  optimizer. The architecture of  is illustrated in Figure <ref>. In the architecture, the green frame denotes the  strategy based , whose detailed structure is provided in Figure <ref>. Next, we will provide more detailed descriptions of both the architecture and the involved learning components. The notations and terminologies we have employed in this paper are presented in Table <ref>. 


 adopts the  strategy to increase the diversity of the input models, which can further resolve the “local sticking” problem of the parent models. The learning process of  involves multiple generations, which can be denoted as G^(1), G^(2),⋯, G^(K) respectively (K is the total generation count). In each generation, g input models will be trained along with the  based  at first; then a batch of trained input models will be selected to compose the parent model pairs; after that, the child models will evolve according to the ; finally, among both the input models and the generated child models, the same number (e.g., g) of new generation models will be selected as the output models of the current generation and as the input models of the next generation. After a predefined number of iterations (or some converging criteria has been satisfied), the training will stop, and the best new-generation models will become the output model. We will refer to each procedure in the following parts.





 §.§ Training Input Models with Boosting Strategy

In each generation G^(k), k ∈{1,2,… ,K},  g input models {M_1, M_2, …, M_g} are trained with  based . More detailed information about the  based  training will be covered in Section <ref>. The essence of  strategy is to train multiple models with collaborating policy <cit.>, then combine these models to induce the final results. For each model, the collaborating is achieved by sampling different sub training sets according to the previous model's performance. Specifically, during the training process of g input models, the same size g training sets {𝒟_1, 𝒟_2, …, 𝒟_g } will be generated. For each training set 𝒟_j, its data samples are sampled from the entire training set, under the distribution determined by the performance of model M_j-1. For our classification task, the data samples having been misclassified by M_j-1 will be assigned higher probabilities to be sampled in 𝒟_j. In this way,  strategy allocates each input model with different training data. Since the different training sets will allow the input model variables to be updated by  differently, these g input models are more likely to converge to different local optimal points. Such a characteristic gives the following  more advantages, and we will discuss this in the later parts. Moreover,  strategy not only allows each input model to learn information hidden in data, but also refers to previously trained models. We also call this process the interaction among the input models. 
After the training process of g input models, {M̅_1, M̅_2,…, M̅_g} refers to the trained input models, and then parent model pairs {(M̅_i_1, M̅_j_1),…, (M̅_i_g, M̅_j_g)}, i, j ∈{1,2,… ,g} will be chosen from them based on special mechanism. Next, the child models will evolve from them by the , which essentially helps the parent models jump out of the local minimum and discover global optima to achieve better learning performance.






 §.§ Parent Model Selection and Fitness Evaluation


Similar to the evolutionary laws in the natural world, where individuals with better performance own higher possibilities to become the parents, our proposed  also inclines to choose input models with better performance to participate in the evolving process. Before the evolving operations, from the input models trained by  based , a batch of parent model pairs will be selected according to the fitness evaluation. The fitness evaluation aims to evaluate each trained input model by computing their fitness scores for the learning setting, and a better fitness score means better performance of the target model. Correspondingly, input models with better fitness scores would be more likely to be selected as the parent models lately. Thus, the function of fitness evaluation is akin to evolutionary laws. 

In the k_th generation G^(k), for each trained input model M̅_i, we calculate its loss on validation set 𝒱 (sampled from the training set) as the fitness score, which can be denoted as follows:

    l[i] = ∑_(X[j,:], y[j]) ∈𝒱 l(X[j,:],y[j]; w_i)

where l(·,·;) is the cross-entropy loss function, w_i denotes the variables of model M̅_i, and vector l∈ℝ^g contains the computed fitness scores of all these g unit models in the current generation. By calculating the loss of input models on the validation set, we can judge these input models: the less the loss is, the better performance model will have in general. However, directly using the loss terms for parent model selection may not work well because the range of l[i] might vary in a large scale. So for each l[i], we utilize the normalized l̂[i] = l[i] - min(l)/max(l) - min(l) to calculate the selecting probability for input model M̅_i to become the parent model as follows: 

    p[i] = exp(-l̂[i])/∑_j=1^gexp(-l̂[j])

where P∈ℝ^g. According to the probability vector p, g different model pairs {(M̅_i_1, M̅_j_1),…, (M̅_i_g, M̅_j_g)} will be sampled as parent models. Specifically, for each model in these pairs, it is probability sampled by p with replacement.




 §.§ Crossover and Mutation


The genetic laws of individuals' genes are common in the natural world to obtain better offspring. 
Inspired by the recombination and mutation rules of genes in genetic laws, the  also carries out similar operations, named crossover and mutation separately. The  includes crossover and mutation operations for creating child models generation, where the model variables are treated as the chromosome of gene, respectively. 

	
  * Crossover: Given a parent model pair (M̅_i_n, M̅_j_n) with respective variable vectors w̅_i_n and  w̅_j_n, crossover generates their child model with variable vector w_n, whose entry w_n[h] can be represented as
	
    w_n[h]    = 1(rand≤ threshold)·w̅_i_n[h] 
       + 1(rand > threshold)·w̅_j_n[h]

	In the Equation <ref>, 1(·) is the indicator function, rand is a random number uniformly distributed in [0,1], threshold = p[i_n]/p[i_n] + p[j_n] and w̅_i_n[h], w̅_j_n[h] denotes the h_th variable of the parent models M̅_i_n and M̅_j_n respectively. By applying the evolving rule in Equation <ref> to each parent model pairs, the model with relatively better performance would retain more its own variables. 
	
  * Mutation: Similar to crossover, the mutation can be expressed as
	
    w_n[h]    = 1(rand≤ p_m)·𝒩(0, 0.01)
       + 1(rand > p_m)·w_n[h]

	where p_m = p·(1 - p[i_n] - p[j_n]) and p is a pre-defined value (e.g., p = 0.01), 𝒩(0, 0.01) is a random number sampled from the normal distribution with 0 mean and 0.01 variance. Equation <ref> can make sure the child models of parents with higher p values to be less likely to mutate. In other words, the mutaton mechanism in Equation <ref> inclines to preserve the variables inherited from parents that having better performance.  

Formally, via crossover and mutation, we can represent the generated g child models as {C_1, C_2, ⋯, C_g}, which will be further trained via the  based . Those trained child models can be denoted as {C̅_1, C̅_2, ⋯, C̅_g} respectively.








 §.§ New Generation Selection

The selection of the new generation of models also follows the evolutionary law: the best and strongest individuals will survive. The new generation models are selected among both the trained child models and input models. In this way, it is certain that the selected new generation of models will not deteriorate compared with the input models. This idea also conforms to the rule of survival of the fittest in natural selection. For each model in {M̅_1,M̅_2,… ,M̅_g}⋃{C̅_1,C̅_2,… ,C̅_g}, its loss on the validation set will be recorded. Finally, we will select the top g models with the smallest losses as the new generation models, which can be denoted as {N_1,N_2,… ,N_g} and they will also serve as the input models for the next generation. Such an iterative model learning and the evolving process will continue until finally converging, and the optimal output model (with the lowest loss on validation set) in the last generation will be selected as the final output model. We will also give the convergence analysis of  in Section <ref> and briefly mention the convergence results in Section <ref>.






 §.§ Boosting Strategy based 

Boosting refers to a family of algorithms that can convert weak learners to strong learners. The core idea of  is to correct the mistakes made by previous learners (models) and let the current model focus more on the data examples being misclassified by prior models. So during the training process of the current model, the training set will be different from those for prior models. In this section, we will talk more about the  based  learning algorithm adopted in .



  §.§.§ Motivation


Prior to adding the  strategy,  is only combined with  to rediscover the best solutions of the models. 

However, we observe that  cannot really resolve such a problem completely. For instance, assume that after training by , all input models converge to the same or close local optima of the loss function, which can be eshibited by Figure <ref>. Under such circumstances, the  (crossover and mutation) can hardly help child models jump out of local optima because the parent models are too similar on the learned model variables. Assume the variables of M̅_i_n has w̅_i_n∈ U(o, δ) = {w |w-o_2 ≤δ}, ∀ i_n ∈{1,2,…, g}, where o represents a local optima and δ is a default small value. Here, U(o, δ) represents a neighborhood region around o in the variable space. Then for parent model pairs M̅_i_n and M̅_j_n located in U(o,δ), we can compute the variables of their child model C_n as w_n[h] = β·w̅_i_n[h] + (1-β)·w̅_j_n[h], where β∈{0,1}, which is exactly the crossover operation in Section <ref>. Then we have

    w_n - o_2    ≤w_n - w̅_i_n_2 + w̅_i_n - o_2 
       ≤w_n - w̅_i_n_2 + δ
       ≤w̅_i_n - w̅_j_n_2 + δ
       ≤w̅_i_n - o_2  + w̅_j_n - o_2 + δ
       ≤ 3δ

and observe that after the crossover, C_n still locates in the neighborhood region of the local optima o. 

This phenomenon is exactly the “local sticking" situation we have mentioned in the Introduction section. It is true that mutation operation can assist in jumping out of the local optima. However, according to Equation (<ref>) the parent models with a relatively small loss on the validation set will lead to a much lower mutation rate p_m for their child models. Thus mutation cannot solve the local sticking problem thoroughly. On the other hand,  can solve this problem by creating different training sets for g input models, which means to let models converge to different local optima. So there will be more gaps and divergence among the learned parent models, which can potentially enhance the advantages of the  and base learners to achieve better solutions. 





  §.§.§ Boosting Strategy


The overall framework of  based  is shown in Figure <ref>. To explicitly explain the  strategy, first, we have to introduce the weight z∈ℝ^m for all the training instances (here, m denotes the size of the total training set). For each input model's training, one sub training set is sampled subject to the current weight vector z. In other words, the value of z[i] represents the probability that i_th training instance will be sampled. Initially, z[i] = 1/m, ∀ i ∈{1, 2, ⋯, m }, i.e., all the instances will be sampled with an equivalent chance. We train the first input model M_1 by  with a sub training set sampled from the entire training set subject to the weight vector z. After this training process, we can denote the trained first model as M̅_1 and record its prediction results on the entire training set to update z by

    z[i] = z[i]×{[ exp(-α_j)         ;;  exp(α_j)         . ].

and

    z[i] = z[i]/ sum(z)

where M̅_j(X[i,:]) denotes the output of model M̅_j on instance X[i,:], α_j = 1/2log(1-ϵ_j/ϵ_j) and y[i] represents the true label of i_th training example. Here, for the first input model M̅_1, j = 1 and ϵ_1 = P_(X[i,:], y[i])∼𝒳(M̅_1(X(i,:)) ≠y[i]). What need to be mentioned is that ϵ_j <0.5 should be satisfied because we deliberately design the weight of training sample X[i,:] having M̅_1(X[i,:]) ≠y[i] to increase, so α_1 will be larger than 0. In this way, sample X[i,:] will be more likely to appear in next sub training set and the next model will focus more on correctly classifying it. The Equation (<ref>) is to regulate z as a probability distribution. When training the next model by , another sub training set will be sampled based on the new weight vector z. By using the weight vector z, the  strategy successfully assigns entire training set with different distributions (redistribution of the training set). 

The  strategy can also be regarded as the interaction among models. By applying  we achieve the interaction among models through updating the sub training set for each input model. What is more, this interactive training essentially makes these g input models diverse. Since different training sets will lead to various loss functions and gradients, g input models are more likely to converge to other local optimal points. This characteristic can give  more advantages, and that is the core idea we implement the  strategy into the .














§ THEORETIC ANALYSIS



In this section, we will show that the proposed  algorithm can guarantee the learning effectiveness; meanwhile, keep converging when the number of generations increases.



 §.§ Effectiveness Guarantee

According to <cit.>, it has been proven that the error ϵ_i of the trained input model M̅_i can be bounded by


    2^g ∏_i = 1^g√(ϵ_i(1 - ϵ_i))≤exp(-2∑_i=1^gγ_i^2)
 
where γ_i = 0.5 - ϵ_i. Let ϵ_min be the error of input model with the smallest loss, we have 


    2^g ϵ^g_min≤ 2^g (ϵ_min(1-ϵ_min))^g/2≤ 2^g ∏_i = 1^g√(ϵ_i(1- ϵ_i))

Therefore ϵ_min≤1/2exp(-2∑_i=1^gγ_i^2/g). The error of the model we finally select as the output model in the last generation has a relatively tight upper bound.






 §.§ Convergence Analysis

In our proposed method, we effectively integrate  based  learning algorithm into  as shown in Figure <ref> and Algorithm <ref>. For each generation G^(k), while training g input models, we no longer use the same training set for every model; instead, the  based learning algorithm shown in Figure <ref> is added to the training process. For M_i, the training set 𝒟_i (denoted by a gray oval) is applied. A remark to be added here, the training process of the child models also applies the  strategy as indicated in the algorithm architecture. With the growth of the k, we will show that the loss of input models can finally converge.


The  algorithm will also converge in a finite number of generations, along with the loss of the input models decreasing continuously. For the i_th input model M_i^(k) in the generation G^(k), the variables learned by  based  will converge after training, which means l̅o̅s̅s̅_i^(k)≤loss_i^(k), where loss_i^(k) and l̅o̅s̅s̅_i^(k) denote the introduced loss of the model M_i and M̅_i before and after training respectively in generation G^(k). Since in the k_th generation,  the top g models are selected among both the trained input models and the generated child models as the output models, the performance of those chosen g models will not be worse than the input models. In other words, loss_i^(k+1)≤l̅o̅s̅s̅_i^(k), ∀ i ∈{1,2,…, g}. Therefore we have 


    loss^(k+1) = ∑_i=1^gloss_i^(k+1)≤∑_i=1^gl̅o̅s̅s̅_i^(k)≤∑_i=1^gloss_i^(k)
    =loss^(k)


With the training process going on, the loss of models in each generation will continuously decrease when K goes up, and the  will finally converge. The convergence results will be exhibited in Section <ref>.




§ NUMERICAL EXPERIMENTS



To test the effectiveness and advantages of the proposed  algorithm, extensive experiments have been conducted on real-world datasets. In this section, we will first describe the datasets we have used in the experiment, and then introduce the experimental settings in detail. Finally, we will exhibit the experimental results together with detailed descriptions and give the parameter sensitivity analysis.



 §.§ Dataset Description


	
  * ORL Dataset: The ORL <cit.> dataset consists of face images of 40 people, each person has ten images. Each image is in size of 112×92.
	
  * MNIST Dataset: The MNIST <cit.> dataset includes 60,000 training samples and 10,000 testing samples, where each sample is a 28×28 image of hand-written numbers from 0 to 9. 
	
  * CIFAR-10 Dataset: The CIFAR-10 <cit.> dataset consists of 60000 32×32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. The dataset has no augmentation operation.






 §.§ Experiment Settings

In this part, we will introduce the experiment settings, which covers the detailed experiment setup, comparison methods, and the evaluation metrics.


  §.§.§ Experiment Setup


We use the convolutional neural network (CNN) structure models as the base model (input model). The CNN model we have built is based on the LeNet-5 <cit.>, which has seven layers. 
For different datasets, the CNN models have different settings:

for the ORL dataset, we use two convolutional layers with 16 and 36 feature maps of 5× 5 kernels and 2× 2 max-pooling layers, and a fully connected layer with 1024 neurons; for the MNIST dataset, we use LeNet-5 structure in CNN model; for the CIFAR-10 dataset, we apply three convolutional layers with 64, 128, 256 kernels respectively, and a fully connected layer having 1024 neurons.

All the experiments apply Relu <cit.> activation function, and 0.5 dropout rate on fully connected layers. 

For each training process, the training batch involves 128 samples and the number of epochs satisfies traversing the entire training set around 200 times. The initialization of variables is random numbers sampled from Normal Distribution with 0 means and 0.01 standard variance. 



For different datasets we use different g and K in the , and we will analyze their influence later.










  §.§.§ Comparison Methods and Evaluation Metrics

To show the advantages of the  algorithm, we compare it with the most commonly used optimization algorithms, including  <cit.>, RMSProp <cit.>, AdaGrad <cit.> and  <cit.> respectively.The  input model in our experiments is based on the CNN structures.

To measure the performance of the comparison methods, different metrics have been applied in this paper. We calculate both the accuracy of prediction and test loss achieved by the models trained with these different optimization algorithms on the test set.







 §.§ Experimental Results

In this section, firstly, we will exhibit results on all the datasets, then show the convergence and hyper-parameter analysis results of the  algorithm.



  §.§.§ Results on All Datasets

In Table <ref>, we show the performance of  algorithm on several datasets compard with other baseline algorithms. The results of  in the table is with g =5, K =5. To let the total training iterations of  be identical to comparison methods (e.g., ), we set the training iterations in each generation of  as the training iterations of  dividing K. 

In this way, the total number of training iterations of  is equal to , meanwhile the overfitting problem can be potentially inhibited. From the table, we can see that the  achieves better overall performance on the test set. For the accuracy of prediction,  achieves the best results on most of the datasets, especially on the CIFAR-10 dataset. The test accuracy achieved by  is 0.6358, which is at least 2.5% larger than the accuracy obtained by ,  and , and the advantages are much more significant compared with ; for the loss on the test set, the advantages of  are much more obvious: the 's results are less than  and  by almost 50 percent on all datasets. Especially for the ORL and CIFAR-10 datasets, the test loss of  is only half of . The advantages are much more significant when comparing to other algorithms such as  and . The overall results demonstrate that our proposed  method does improve the performance of both  and  by adopting the  strategy.







  §.§.§ Convergence and Hyper-Parameter Analysis

 algorithm can converge in a finite number of generations, which is shown in Figure <ref>. The result is on the ORL-7 dataset.

Due to the limited space, we will only show the analysis on the ORL dataset in this part. The hyper-parameters g and K involved in the training process may affect the convergence and final results of . Thus we also analyze their influence. We run the experiments with g ∈{1,2,…,10} and K∈{1,2,…,10} on the ORL dataset to illustrate their influence. The results are shown in Figure <ref>. 
We can notice that the test loss decreases when K increases in terms of different g numbers, and conversely, the accuracy on the test set increases. The trend can also be seen in Table <ref>.



We also find that the loss function does not change when K and g increase from 5 to 10, respectively. In other words, the proposed  algorithm can converge to an ideal solution in 5 generations with the input models size of 5 generally. That is also why we apply the setting of g = 5 and K = 5 when carrying out the final results of . For the experiments on the other datasets, we also check the influence of K and verify specific values of K to get the final model.












§ CONCLUSION



In this paper, we have introduced a new hybrid optimization algorithm, namely . By combining , , and  strategy together,  can maximize the advantages of each part by utilizing their characteristics to efficiently jump out of local optima and prevent the “local sticking” phenomenon, then further converge to better solutions. We have carried out extensive experiments on real-world datasets, and the results show that our proposed  algorithm outperforms previous optimization methods, especially for training deep neural networks.





§ ACKNOWLEDGEMENT



This work is partially supported by NSF through grant IIS-1763365.






plain













