<p><strong>Summary of your results:</strong></p>
<ul>
<li>a) CNN with Softmax activation function -&gt; accuracy ~ 0.50, loss ~ 7.60</li>
<li>b) CNN with Sigmoid activation function -&gt; accuracy ~ 0.98, loss ~ 0.06</li>
</ul>
<p><strong>TLDR</strong></p>
<p>Update:</p>
<p>Now that I also see you are <strong>using only 1 output neuron with Softmax, you will not be able to capture the second class</strong> in binary classification. <strong>With Softmax you need to define K neurons in the output layer</strong> - where K is the number of classes you want to predict. Whereas with Sigmoid: 1 output neuron is sufficient for binary classification.</p>
<p>so in short, this should change in your code when using softmax for 2 classes:</p>
<pre><code>#use 2 neurons with softmax
model.add(tf.keras.layers.Dense(2, activation='softmax'))
</code></pre>
<p>Additionally:</p>
<p>When doing <strong>binary classification</strong>, a <strong>sigmoid function is more suitable</strong> as it is simply <strong>computationally more effective</strong> compared to the more generalized softmax function (which is normally being used for multi-class prediction when you have K&gt;2 classes).</p>
<hr />
<p><strong>Further Reading:</strong></p>
<p><strong>Some attributes of selected activation functions</strong></p>
<p>If the short answer above is not enough for you, I can share with you some things I've learned from my research about activation functions with NNs in short:</p>
<p>To begin with, let's be clear with the terms activation and activation function</p>
<blockquote>
<p>activation (alpha): is the state of a neuron. The state of neurons in hidden or output layers will be quantified by the weighted sum of input signals from a previous layer</p>
</blockquote>
<blockquote>
<p>activation function f(alpha): Is a function that transforms an activation to a neuron signal. Usually a non-linear and differentiable function as for instance the sigmoid function. Many applications &amp; research has been applied with the sigmoid function (see Bengio &amp; Courville, 2016, p.67 ff.). Mostly the same activation function is being used throughout the neural network, but it is possible to use multiple (e.g. different ones in different layers).</p>
</blockquote>
<p>Now to the effects of activation functions:</p>
<blockquote>
<p>The choice of activation function can have an immense impact on learning of neural networks (as you have seen in your example). Historically it was common to use the sigmoid function, as it was a good function to depict a saturated neuron. Today, especially in CNNs other activation functions, also only partially linear activation functions (like relu) is being preferred over sigmoid function. There are many different functions, just to name some: sigmoid, tanh, relu, prelu, elu ,maxout, max, argmax, softmax etc.</p>
</blockquote>
<p>Now let's only compare sigmoid, relu/maxout and softmax:</p>
<pre><code># pseudo code / formula
sigmoid = f(alpha) = 1 / (1 + exp(-alpha))
relu = f(alpha) = max(0,alpha)
maxout = f(alpha) = max(alpha1, alpha2)
softmax = f(alpha_j) = alpha_j / sum_K(alpha_k)
</code></pre>
<p>sigmoid:</p>
<ul>
<li>in binary classification preferably used for output layer</li>
<li>values can range between [0,1], suitable for a probabilistic interpretation (+)</li>
<li>saturated neurons can eliminate gradient (-)</li>
<li>not zero centered (-)</li>
<li>exp() is computationally expensive (-)</li>
</ul>
<p>relu:</p>
<ul>
<li>no saturated neurons in positive regions (+)</li>
<li>computationally less expensive (+)</li>
<li>not zero centered (-)</li>
<li>saturated neurons in negative regions (-)</li>
</ul>
<p>maxout:</p>
<ul>
<li>positive attributes of relu (+)</li>
<li>doubles the number of parameters per neuron, normally requires an increased learning effort (-)</li>
</ul>
<p>softmax:</p>
<ul>
<li>can bee seen as a generalization of sigmoid function</li>
<li>mainly being used as output activation function in multi-class prediction problems</li>
<li>values range between [0,1], suitable for a probabilistic interpretation  (+)</li>
<li>computationally more expensive because of exp() terms (-)</li>
</ul>
<p>Some good references for further reading:</p>
<ul>
<li><a href="http://cs231n.stanford.edu/2020/syllabus" rel="nofollow noreferrer">http://cs231n.stanford.edu/2020/syllabus</a></li>
<li><a href="http://deeplearningbook.org" rel="nofollow noreferrer">http://deeplearningbook.org</a> (Bengio &amp; Courtville)</li>
<li><a href="https://arxiv.org/pdf/1811.03378.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1811.03378.pdf</a></li>
<li><a href="https://papers.nips.cc/paper/2018/file/6ecbdd6ec859d284dc13885a37ce8d81-Paper.pdf" rel="nofollow noreferrer">https://papers.nips.cc/paper/2018/file/6ecbdd6ec859d284dc13885a37ce8d81-Paper.pdf</a></li>
</ul>
Activation functions: Softmax vs Sigmoid
<p>I've been trying to build an image classifier with CNN. There are 2300 images in my dataset and two categories: men and women. Here's the model I used:</p>
<pre><code>early_stopping = EarlyStopping(min_delta = 0.001, patience = 30, restore_best_weights = True)
model = tf.keras.Sequential()

model.add(tf.keras.layers.Conv2D(256, (3, 3), input_shape=X.shape[1:],  activation = 'relu'))

model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv2D(256, (3, 3), input_shape=X.shape[1:], activation = 'relu'))
model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Flatten())  # this converts our 3D feature maps to 1D feature vectors

model.add(tf.keras.layers.Dense(64))

model.add(tf.keras.layers.Dense(1, activation='softmax'))


model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

h= model.fit(xtrain, ytrain, validation_data=(xval, yval), batch_size=32, epochs=30, callbacks = [early_stopping], verbose = 0)
</code></pre>
<p>Accuracy of this model is 0.501897 and loss 7.595693(the model is stuck on these numbers in every epoch) but if I replace Softmax activation with Sigmoid, accuracy is about 0.98 and loss 0.06. Why does such strange thing happen with Softmax? All info I could find was that these two activations are similar and softmax is even better but I couldn't find anything about such abnormality. I'll be glad if someone could explain what the problem is.</p>