<p><code>Q1: Why my network is still trainable with a *wrong* loss function?</code></p>

<p>A1: Because your network is optimized in terms of gradient descent, which does not care about which loss function is used as long as it is differentiable. This fact reveals the difficulty to debug a network when it doesn't work, because it is not a code bug (e.g. causing memory leak, numerical overflow, etc.), but some bug does not scientifically sound (e.g. your regression target is of range (0,100), but you use <code>sigmoid</code> as the activation function of the last dense layer).</p>

<pre><code>Q2: How come `sigmoid` gives better performance than `softmax`?
</code></pre>

<p>A2: First, using the <code>sigmoid</code> loss function means to train 10 binary classifiers, one for each class (i.e. the classic one v.s. all or one v.s. rest setting), and thus it is also technically sound. </p>

<p>The only difference between <code>sigmoid</code> and <code>softmax</code> is that the sum of the class-wise predicted probability is always 1 for the <code>softmax</code> network, while may not necessarily to be 1 for the <code>sigmoid</code> network. In other words, you might have confusions to decide a label during testing for the <code>sigmoid</code> network. </p>

<p>Regarding to why <code>sigmoid</code> is better than <code>softmax</code>, it is related to many aspects and difficult to analyze without careful studies. One possible explanation is that <code>sigmoid</code> treats rows in the weight matrix of the last dense layer independently, while <code>softmax</code> treats them dependently. Therefore, <code>sigmoid</code> may better handle those samples with contradicting gradient directions. Another thought is that maybe you should try the recent <a href="https://arxiv.org/pdf/1809.04157.pdf" rel="nofollow noreferrer">heated-up softmax</a>.</p>

<p>Finally, if you believe <code>sigmoid</code> version gives you better performance but you still want a <code>softmax</code> network, you may reuse all the layers until the last dense layer in the <code>sigmoid</code> network and finetune a new <code>softmax</code> layer, or use both losses just as in a multi-task problem.</p>
Using sigmoid activation function for multi-class prediction
<p>I am running an U-net as defined below: </p>

<pre><code>inputs = Input((IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS))
s = Lambda(lambda x: x / 255) (inputs)

c1 = Conv2D(8, (3, 3), activation='relu', padding='same') (s)
c1 = Conv2D(8, (3, 3), activation='relu', padding='same') (c1)
p1 = MaxPooling2D((2, 2)) (c1)

c2 = Conv2D(16, (3, 3), activation='relu', padding='same') (p1)
c2 = Conv2D(16, (3, 3), activation='relu', padding='same') (c2)
p2 = MaxPooling2D((2, 2)) (c2)

c3 = Conv2D(32, (3, 3), activation='relu', padding='same') (p2)
c3 = Conv2D(32, (3, 3), activation='relu', padding='same') (c3)
p3 = MaxPooling2D((2, 2)) (c3)

c4 = Conv2D(64, (3, 3), activation='relu', padding='same') (p3)
c4 = Conv2D(64, (3, 3), activation='relu', padding='same') (c4)
p4 = MaxPooling2D(pool_size=(2, 2)) (c4)

c5 = Conv2D(128, (3, 3), activation='relu', padding='same') (p4)
c5 = Conv2D(128, (3, 3), activation='relu', padding='same') (c5)

u6 = Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same') (c5)
u6 = concatenate([u6, c4])
c6 = Conv2D(64, (3, 3), activation='relu', padding='same') (u6)
c6 = Conv2D(64, (3, 3), activation='relu', padding='same') (c6)

u7 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c6)
u7 = concatenate([u7, c3])
c7 = Conv2D(32, (3, 3), activation='relu', padding='same') (u7)
c7 = Conv2D(32, (3, 3), activation='relu', padding='same') (c7)

u8 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c7)
u8 = concatenate([u8, c2])
c8 = Conv2D(16, (3, 3), activation='relu', padding='same') (u8)
c8 = Conv2D(16, (3, 3), activation='relu', padding='same') (c8)

u9 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c8)
u9 = concatenate([u9, c1], axis=3)
c9 = Conv2D(8, (3, 3), activation='relu', padding='same') (u9)
c9 = Conv2D(8, (3, 3), activation='relu', padding='same') (c9)

outputs = Conv2D(10, (1, 1), activation='sigmoid') (c9)

model = Model(inputs=[inputs], outputs=[outputs])

model.compile(optimizer='Adamax', loss = dice, metrics = [mIoU])
</code></pre>

<p>Notice that I'm doing multi-class prediction on ten classes. And the inputs are <code>256x256x3</code> (rgb) images and the ground truths are binary masks of size <code>256x256x10</code> since the <code>depth=num_classes=10</code>. My question is, I accidently forgot to change the activation function from <code>sigmoid</code> to <code>softmax</code> and ran the network. The network still ran. How is this possible?? Is it because it's treating each binary mask independently? </p>

<p>More intriguingly, the network actually yielded better results when using <code>sigmoid</code> as opposed to when I ran it with <code>softmax</code>.</p>