value will have little effect. In addition, a larger LR with decay and a larger momentum are suggested because fewer neurons updated with dropout requires more update for each batch [@brownlee16].\
Activation functions are crucial in deep learning for introducing nonlinear properties to the output of neurons. Without an activation function, a neural network will simply be a linear regression model that is unable to represent complicated features of data. Activation functions should be differentiable to compute the gradients of weight and perform backpropagation[@walia17]. The most popular and widely used activation functions include sigmoid, hyperbolic tangent (tanh), rectified linear units (ReLU) [@nair10], Maxout [@goodfellow13], and Swish [@ramachandran17a]. Automatic search techniques have been applied in search of proper activation functions, including the structure of functions and related hyper-parameters [@ramachandran17b].\
Sigmoid functions are the most frequently used activation layer with a form of $f(x)\;=\; \frac{1}{1+e^{-x}}$. The output of a sigmoid function is always between 0 and 1 (Figure \[Figure 8\]a) when negative infinite to positive infinite are encountered, and thus it will not blow up the activation. They provide nonlinearity and make clear distinctions, but in the past few years they have fallen out of popularity because of some major disadvantages. With 