title: Accelerating training in Multilabel classification with NNs
body: <p>I am using Tensorflow for multi-label classification of Audio. The dataset I am using is made up of 10 different classes, and to each sample of audio correspond two labels. In other words, the number of overlapping sounds/classes per file is always 2. So the vector of labels "y" would be for example: </p>
  
  <p>y = [0, 0, 0, 0, 0, 1, 0, 1, 0, 0]</p>
  
  <p>I am using 3 convolutional layers , 1 FC layer and 1 output layer made up of 10 logistic units. And as cost function I am using the binary cross entropy for each unit and then averaging the result.</p>
  
  <p>For some reason(s), I guess mainly as a consequence of the initialization of the parameters (weights) of the network and the logistic units in the output layer, the network tends to predict mainly zeros. I guess after some training this trend will start to dissapear, however, I wonder if there is a smart way of doing something so the network does not <strong>predict those many zeros and predicts more ones</strong>, which I believe would make the cost higher and the gradient stronger (not 100% sure about this one though)  and therefore training would be faster. </p>
  
  <p>I use Xavier initialization for the weights of all layers, as an example:</p>
  
  <pre><code>W2 = tf.get_variable("W", [5,5,24,48], initializer=tf.contrib.layers.xavier_initializer())
  </code></pre>
  
  <p>Maybe using <code>tf.truncated_normal_initializer</code> instead
  could help? </p>
  
  <p>----------------------------------------- EDIT 1 --------------------------------------------------------</p>
  
  <p>A piece of information that I forgot to mention is that all my inputs are negative values, I don't know if that actually has any consequences. </p>