<p>When doing binary classification, using <a href="http://caffe.help/manual/layers/softmaxwithloss.html" rel="nofollow noreferrer"><code>"SoftmaxWithLoss"</code></a> with two outputs, is mathematically equivalent to using <a href="http://caffe.help/manual/layers/sigmoidcrossentropyloss.html" rel="nofollow noreferrer"><code>"SigmoidCrossEntropyLoss"</code></a>. So, if you really only need one output you can set your last layer to <code>num_output: 1</code> and use <code>"SigmoidCrossEntropyLoss"</code>. However, if you want to take advantage of caffe's <code>"Accuracy"</code> layer, you need to use two outputs and <code>"SoftmaxWithLoss"</code> layer.  </p>

<p>Regarding your questions:<br>
1. If you opt to use <code>"SoftmaxWithLoss"</code> and you only need one output, take the second output for each pixel as this entry represents the probability of class 1.<br>
I'll leave it to you as an exercise to figure out what you'll get if you take the sum (hint: `"Softmax" output probabilities...)<br>
2. The loss is very small most likely because you have severe class imbalance - most of your pixels are 0 while only very few are 1 (or vice versa), therefore predicting always 0 does not incur such great penalty. If this is your case, I suggest looking at <a href="https://arxiv.org/abs/1708.02002" rel="nofollow noreferrer">Focal Loss</a> that addresses this issue.</p>
Interpret Caffe FCN output classes
<p>I have set up Caffe and using FCN-8s model with little change with output classes:</p>

<pre><code>layer {
 name: "score_5classes"
 type: "Convolution"
 bottom: "score"
 top: "score_5classes"
 convolution_param {
    num_output: 2
    pad: 0 
    kernel_size: 1 
    }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "score_5classes"
  bottom: "label"
  top: "loss"
  loss_param {
    normalize: true
  }
}
</code></pre>

<p>I have changed last layer output number to 2, because I want to classify my input images into 2 classes, 0 and 1 (So it seems I should have 2 outputs! I cant understand why?! It could be an output matrix with zeros and ones, couldnt it?)</p>

<p>So my questions are:</p>

<p>1.Should I sum these 2 classes ? because I need 1 output</p>

<p>2.The loss is so small! even when the output is far away from the desired! how Caffe calculates the lost layer?</p>

<p>Thanks</p>