sigmoid function, one finds that the output f(x) responds very little to the input x, which gives rise to the vanishing gradient problem [@wang2019vanishing]. This may not be a big problem for networks with a few layers, but it causes weights that are too small when more layers are required in a network. Another problem is it not being zero-centered, which means the average of data lies on the zero. During backpropagation, weights will either be all positive or all negative, making an undesirable zigzag on weights and making optimization more difficult. Softmax is a function similar to a sigmoid function with each input within (0, 1). The major difference is that a sigmoid function is applied for binary classification, whereas a softmax function is for a multinomial classification. With j classes, the predicted probability of $i^{th}$ class is as $f(x_i)\;=\; \frac{exp(x_i)}{\sum jexp(x_i)}$. In addition, softmax function allows a probabilistic interpretation. With the softmax function, a high input value results in a higher probability than others, and the sum of probabilities will be 1. Whereas with the sigmoid function, a high value will lead to the high probability as well, but not the higher probability, and the sum of probabilities 