is not necessarily to be 1. While creating neural networks, sigmoid function is used as activation function, and softmax function is usually used in output layers.\
Hyperbolic tangent (tanh) is in a form of $f(x)\;=\; \frac{1-e^{-2x}}{1+e^{-2x}}$. It is zero-centered and the output ranges between minus one and one (Figure \[Figure 8\]b). It is in a similar structure to a Sigmoid function but with a steeper derivative. Tanh is more efficient than a Sigmoid function and has a wider range for faster learning and grading [@kizrak19], but it still has the problem of vanishing gradients. Sigmoid and tanh cannot be applied for networks with many layers.\

![Comparison of activation functions: (a) sigmoid; (b) tanh; (c) ReLU; and (d) Swish []{data-label="Figure 8"}](8.png){width="13.5cm" height="8.5cm"}

In recent years, ReLU has become the most widely used activation function [@agarap18]. More advanced activation functions have arisen but ReLU and its variants (e.g., leaky ReLU, PReLU, ElU, and SeLU) are still the most typical and popular methods (Figure \[Figure 8\]c).\
$$f(x)\;=\;\left\{
        \begin{array}{rcl}
        0, & x\;\leq\;0\\
        x, & x\;> \;0\\
        \end{array}
        \right.$$ ReLU is mathematically simple and solves the problem of vanishing gradients. Compared with sigmoid and tanh, ReLU does not covert almost all neurons to be activated 