& SGD & RMS & Adam & eSGD & aSGD & Ours & SGD & RMS & Adam & eSGD & aSGD & Ours\
ave & 89.23 & 88.89 & 88.98 & 87.63 & 89.12 & **89.49** & 89.71 & 89.17 & 89.26 & 88.13 & 89.24 & **89.95**\
max & 89.50 & 89.15 & 89.28 & 87.83 & 89.44 & **89.84** & 89.87 & 89.54 & 89.49 & 88.32 & 89.56 & **90.23**\
& &\
& SGD & RMS & Adam & eSGD & aSGD & Ours & SGD & RMS & Adam & eSGD & aSGD & Ours\
ave & 90.65 & 90.81 & 90.78 & 89.76 & 90.12 & **90.87** & 93.45 & 91.97 & 92.08 & 93.33 & 93.05 & **93.51**\
max & 91.36 & 91.30 & 91.23 & 90.54 & 90.48 & **91.51** & 93.73 & 92.36 & 92.46 & 93.72 & 93.37 & **93.83**\

\
(3) Validation accuracy for CIFAR-10\

[l | PPPPPP | PPPPPP]{} & &\
& SGD & RMS & Adam & eSGD & aSGD & Ours & SGD & RMS & Adam & eSGD & aSGD & Ours\
ave & 94.70 & 90.72 & 90.92 & 91.46 & 93.07 & **94.80** 