5"}](5.png)

Root mean square prop (RMSprop) is one of the most widely used optimizers in the training of deep neural networks [@karpathy16]. RMSprop accelerates the gradient descent in a manner similar to Adagrad and Adadelta [@zeiler12], but it exhibits superior performance when steps become smaller. It can be considered the development of Rprop [@igel00] for mini-batch weight update. In RMSprop, the LR is adapted by dividing by the root of the squared gradient. Compared with the original GD, RMSprop slows the vertical oscillation and accelerates the horizontal movement from a much larger LR [@bushaev18], and it is implemented as follows. RMSprop uses exponential weighted averages of the squares instead of directly using *dw* and *db*:\
$${S_{dw}} \;=\;\beta{S_{dw}}+(1-\beta)dw^2$$ $${S_{db}} \;=\;\beta{S_{db}}+(1-\beta)db^2$$ $$w \;=\;w-lr*\frac{dw}{sqrt(S_{dw})}$$ where $\beta$ is the moving average parameter, suggested as 0.9 [@hinton12b]. *$S_{dw}$* is relatively small and thus *dw* is divided by a relatively small number, whereas *$S_{db}$* is relatively large and thus *db* s divided by a larger number. In this way, RMSprop endures a slower movement in the bias direction (vertical) and a faster move in the weight direction (horizontal; Figure 6).\

![Comparison of SGD (orange) and RMSprop (green) for optimization[]{data-label="Figure 6"}](6.png)

Adaptive momentum estimation (Adam) also achieves 