<p>Why don't you use the <a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1InfogainLossLayer.html" rel="noreferrer"><strong>InfogainLoss</strong></a> layer to compensate for the imbalance in your training set?</p>
<p>The Infogain loss is defined using a weight matrix <code>H</code> (in your case 2-by-2) The meaning of its entries are</p>
<pre><code>[cost of predicting 1 when gt is 0,    cost of predicting 0 when gt is 0
 cost of predicting 1 when gt is 1,    cost of predicting 0 when gt is 1]
</code></pre>
<p>So, you can set the entries of <code>H</code> to reflect the difference between errors in predicting 0 or 1.</p>
<p>You can find how to define matrix <code>H</code> for caffe in <a href="https://stackoverflow.com/questions/27632440/infogain-loss-layer">this thread</a>.</p>
<p>Regarding sample weights, you may find <a href="http://deepdish.io/2014/11/04/caffe-with-weighted-samples/" rel="noreferrer">this post</a> interesting: it shows how to modify the <strong>SoftmaxWithLoss</strong> layer to take into account sample weights.</p>
<hr />
<p>Recently, a modification to cross-entropy loss was proposed by <a href="https://arxiv.org/abs/1708.02002" rel="noreferrer"><em>Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár</em> <strong>Focal Loss for Dense Object Detection</strong>, (ICCV 2017)</a>.<br />
The idea behind focal-loss is to assign different weight for each example based on the relative difficulty of predicting this example (rather based on class size etc.). From the brief time I got to experiment with this loss, it feels superior to <code>&quot;InfogainLoss&quot;</code> with class-size weights.</p>
Tackling Class Imbalance: scaling contribution to loss and sgd
<p><strong>(An update to this question has been added.)</strong></p>

<p>I am a graduate student at the university of Ghent, Belgium; my research is about emotion recognition with deep convolutional neural networks. I'm using the <a href="http://caffe.berkeleyvision.org/" rel="noreferrer">Caffe</a> framework to implement the CNNs.</p>

<p>Recently I've run into a problem concerning class imbalance. I'm using 9216 training samples, approx. 5% are labeled positively (1), the remaining samples are labeled negatively (0).</p>

<p>I'm using the <a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SigmoidCrossEntropyLossLayer.html" rel="noreferrer">SigmoidCrossEntropyLoss</a> layer to calculate the loss. When training, the loss decreases and the accuracy is extremely high after even a few epochs. This is due to the imbalance: the network simply always predicts negative (0). <em>(Precision and recall are both zero, backing this claim)</em></p>

<p>To solve this problem, I would like to <strong>scale the contribution to the loss depending on the prediction-truth combination</strong> (punish false negatives severely). My mentor/coach has also advised me to <strong>use a scale factor when backpropagating</strong> through stochastic gradient descent (sgd): the factor would be correlated to the imbalance in the batch. A batch containing only negative samples would not update the weights at all.</p>

<p><em>I have only added one custom-made layer to Caffe: to report other metrics such as precision and recall. My experience with Caffe code is limited but I have a lot of expertise writing C++ code.</em></p>

<hr>

<p><strong>Could anyone help me or point me in the right direction on how to adjust the <a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SigmoidCrossEntropyLossLayer.html" rel="noreferrer">SigmoidCrossEntropyLoss</a> and <a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1SigmoidLayer.html" rel="noreferrer">Sigmoid</a> layers to accomodate the following changes:</strong></p>

<ol>
<li>adjust the contribution of a sample to the total loss depending on the prediction-truth combination (true positive, false positive, true negative, false negative).</li>
<li>scale the weight update performed by stochastic gradient descent depending on the imbalance in the batch (negatives vs. positives).</li>
</ol>

<p>Thanks in advance!</p>

<hr>

<h2>Update</h2>

<p>I have incorporated the <strong><a href="http://caffe.berkeleyvision.org/doxygen/classcaffe_1_1InfogainLossLayer.html#details" rel="noreferrer">InfogainLossLayer</a> as suggested by <a href="https://stackoverflow.com/a/30497907/1714410">Shai</a></strong>. I've also added another custom layer that builds the infogain matrix <code>H</code> based on the imbalance in the current batch.</p>

<p>Currently, the matrix is configured as follows:</p>

<pre><code>H(i, j) = 0          if i != j
H(i, j) = 1 - f(i)   if i == j (with f(i) = the frequency of class i in the batch)
</code></pre>

<p><em>I'm planning on experimenting with different configurations for the matrix in the future.</em></p>

<p>I have tested this on a 10:1 imbalance. The results have shown that the network is learning useful things now: <em>(results after 30 epochs)</em></p>

<ul>
<li>Accuracy is approx. ~70% (down from ~97%);</li>
<li>Precision is approx. ~20% (up from 0%);</li>
<li>Recall is approx. ~60% (up from 0%).</li>
</ul>

<p>These numbers were reached at around 20 epochs and didn't change significantly after that.</p>

<p><em>!! The results stated above are merely a proof of concept, they were obtained by training a simple network on a 10:1 imbalanced dataset. !!</em></p>