with a sigmoid function that starts from $\eta=0.1$ and ends at $\eta=0.001$ is applied for SGD, eSGD, aSGD, and AdaDecay based on a pre-experiment result in which SGD with the sigmoid learning-rate annealing has achieved better accuracy than those with the fixed learning rate, exponential function, and staircase. We use grid search and set 0.95 as the weighting factor in RMSprop, 0.9 and 0.999 for the first and second momentum factors in Adam. The learning-rate scale in RMS and Adam is set as 0.001 for the shallow networks, and 0.0001 for the deep networks. The hyper-parameters for eSGD and aSGD are also set as the recommended in the original papers [@Chaudhari2017EntropySGD; @Kidambi2018Acc] including the Langevin loop number of 5 for eSGD.

We perform the training process for 100 epochs, and use the maximum and the last $10\%$-epoch mean of the validation accuracy for the test data as the evaluation measures of each trial. For quantitative comparison, we repeat the training process of the shallow networks with MNIST and Fashion-MNIST datasets for 50 independent trials, and the deep networks with CIFAR-10 dataset for 32 trials. We consider both the maximum of the validation accuracy across epochs and trials, and the $10\%$-trimmed 