model oscillates within a small range but does not exactly converge in the last few steps (Figure \[Figure 4\]). An opinion exists that for enhanced accuracy, mini-batch size and LR could be re-optimized after other hyper-parameters are fixed [@masters18]. Vanilla mini-batch GD without momentum may take longer than some more recent optimizers, and the convergence relies on a robust initial value and LR. As the foundation of more advanced optimization algorithms, mini-batch GD is still used but not widely in recent publications [@ruder16].\
An improved method for solving the problem of oscillation and convergence speed is SGD with momentum [@loizou17]. The momentum method [@qian99] accelerates the standard SGD by calculating the exponentially weighted averages for gradients (Figure \[Figure 5\]). Furthermore, it helps the cost function go in the correct direction through adding a fraction beta of the update vector: $${v_{dw}} \;=\;\beta{v_{dw}}+(1-\beta)dw$$ $$w \;=\;w-lr*{v_{dw}}$$ where *w* indicates the weight. Here, momentum $\beta$ is also a hyper-parameter. The momentum term is usually set as 0.9, or 0.99 or 0.999 if necessary. This reduces the oscillation by strengthening the update in the same direction and decreasing the change in different directions [@sutskever13].\

![(a) SGD without momentum; (b) SGD with momentum (Source: https://www.willamette.edu/ gorr/classes/cs449/momrate.html)[]{data-label="Figure 