technique that is in particular used with classical shallow networks. The dropout zeros the activation of randomly selected nodes with a certain probability during the training process [@srivastava2014dropout]. The dropping rate is generally set to be constant but its variants have been considered with adaptive rates depending on parameter value [@ba2013adaptive], estimated gradient variance [@kingma2015variational], biased gradient estimator [@srinivas2016generalized], layer depth [@huang2016deep], or marginal likelihood over noises [@noh2017regularizing]. However, in fact, recent deep models do not support the dropout and its variants. The reason may be that the number of parameters in a layer is relatively smaller than the the classic neural networks, and random masking to nodes can be erroneous to the model.

[**Energy Landscape:**]{} The geometrical property of energy surface is helpful in optimization of highly complex non-convex problems associated with deep network architecture. It is preferred to drive a solution toward local minima on a flat energy surface that is considered to yield better generalization [@hochreiter1997flat; @Chaudhari2017EntropySGD; @dinh2017sharp] where flatness is defined around the minimum by its connected region, its curvature of the second order structure, and the width of its basin, respectively. A geometry-driven optimization based on SGD has been developed in deep learning problems such 