good results quickly on most neural network architectures. As a combination of GD with momentum and RMSprop, Adam adds bias-correction and momentum to RMSprop, enabling it to slightly outperform RMSprop in the late stage of optimization. It calculates the exponential weighted average as well as squares of past gradients. Parameters *$\beta_1$* and *$\beta_2$* control the decay rate as shown below. Moreover, Adam retains an exponential decaying average of past gradients. Its requirement for memory is in the middle of GD with momentum and RMSprop. In practice, Adam is suggested as the default optimization algorithm for deep learning training [@karpathy16]. It has more hyper-parameters compared with other optimizers, but works well with little tuning of hyper-parameters except for LR.\
*$v_{dw}$* and *$v_{db}$* are updated as with GD with momentum:\
$${v_{dw}} \;=\;{\beta_1}{v_{dw}}+(1-{\beta_1})dw$$ $${v_{db}} \;=\;{\beta_1}{v_{db}}+(1-{\beta_1})db$$ *$S_{dw}$* and *$S_{db}$* are updated as with RMSprop:\
$${S_{dw}} \;=\;{\beta_2}{S_{dw}}+(1-{\beta_2})dw^2$$ $${S_{db}} \;=\;{\beta_2}{S_{db}}+(1-{\beta_2})db^2$$ The biases are corrected as follows: $${v_{dw}^{corrected}} \;=\;{v_{dw}}+(1-{\beta_1^t})$$ $${v_{db}^{corrected}} \;=\;{v_{db}}+(1-{\beta_1^t})$$ $${S_{dw}^{corrected}} \;=\;{S_{dw}}+(1-{\beta_w^t})$$ weights *w* and biases *b* are updated as follows:\
$$w \;=\;w-lr*\frac{v_{dw}^{corrected}}{sqrt(S_{dw}^{corrected}+\epsilon)}$$ $$b \;=\;b-lr*\frac{v_{db}^{corrected}}{sqrt(S_{db}^{corrected}+\epsilon)}$$ For the adjustment of LR, the previous section can be referred to, and it is reasonable to start from 0.001. *$\beta_1$* is the parameter of momentum, with a suggested value of 