RMSprop (yellow), Adam (blue), Engropy-SGD (magenta), and Accelerated-SGD (green) across both the shallow networks with MNIST (1) and Fashion-MNIST (2), and the deep networks with CIFAR-10 (3).

\(1) Validation accuracy for MNIST\

[l | PPPPPP | PPPPPP]{} & &\
& SGD & RMS & Adam & eSGD & aSGD & Ours & SGD & RMS & Adam & eSGD & aSGD & Ours\
ave & 98.53 & 98.22 & 98.19 & 98.16 & 98.15 & **98.56** & 98.66 & 98.31 & 98.26 & 98.29 & 98.23 & **98.69**\
max & 98.63 & 98.36 & 98.31 & 98.31 & 98.35 & **98.72** & 98.80 & 98.49 & 98.47 & 98.41 & 98.45 & **98.82**\
& &\
& SGD & RMS & Adam & eSGD & aSGD & Ours & SGD & RMS & Adam & eSGD & aSGD & Ours\
ave &99.31 & 99.30 & 99.27 & 99.23 & 99.17 & **99.32** & 99.62 & 99.37 & 99.37 & 99.58 & 99.52 & **99.63**\
max &**99.48** & 99.39 & 99.37 & 99.38 & 99.28 & 99.45 & **99.71** & 99.50 & 99.43 & 99.63 & 99.59 & 99.70\

\
(2) Validation accuracy for Fashion-MNIST\

[l | PPPPPP | PPPPPP]{} & &\
