title: Tensorflow : Cross Entropy for Multi labels classification
body: <p>I'm using a neural network for classification of a set of objects into n-classes. Each object can belong to multiple classes at the same time (multi-class, multi-label).</p>
  
  <p>For example :</p>
  
  <pre><code>Input is ['a','b','c']  and labels are  [1,0,0,1,1,1]  # where 1 means it is in class and 0 means it's not in class  it's multi labels classification
  </code></pre>
  
  <p>I search and found people are suggesting you should use sigmoid instead of softmax in Multi-label problems.</p>
  
  <p>some references are :</p>
  
  <p><a href="https://stackoverflow.com/questions/45004514/how-to-classify-both-sentiment-and-genres-from-movie-reviews-using-cnn-tensorflo/45005490?noredirect=1#comment87586644_45005490">Sentiment and genres classification </a></p>
  
  <p><a href="https://www.reddit.com/r/MLQuestions/comments/5g3got/how_can_i_use_tensorflow_image_classifier_to_do/" rel="nofollow noreferrer">reddit group suggestion on multi label classification</a></p>
  
  <p><a href="https://nickcdryan.com/2017/01/23/multi-label-classification-a-guided-tour/" rel="nofollow noreferrer">Excellent blog on explaining the theory of multi label classification</a></p>
  
  <p><a href="https://www.dlology.com/blog/how-to-do-multi-class-multi-label-classification-for-news-categories/" rel="nofollow noreferrer">another good tutorial on multi label</a></p>
  
  <p><a href="https://stats.stackexchange.com/questions/207794/what-loss-function-for-multi-class-multi-label-classification-tasks-in-neural-n">Stack exchange question</a></p>
  
  <p>Now after reading those things , I got it that i should go for sigmoid butI have few confusions :</p>
  
  <p>so for single classification what i was doing :</p>
  
  <p>After model (hiddenlayers with lstm )  i was building fully connected layer :</p>
  
  <pre><code>    #weights
      Wo = tf.get_variable('Wo', 
                      shape=[hdim*2, num_labels], 
                      dtype=tf.float32, 
                      initializer=tf.random_uniform_initializer(-0.01, 0.01))
  
  
     #bias
     bo = tf.get_variable('bo', 
                      shape=[num_labels,], 
                      dtype=tf.float32, 
                      initializer=tf.random_uniform_initializer(-0.01, 0.01))
  
  
    #logits
    logits = tf.matmul(tf.concat([fsf.c, fsb.c], axis=-1), Wo) + bo
  
  
     #probability 
     probs  = tf.nn.softmax(logits)
  
     #taking max probability
     preds  = tf.argmax(probs, axis=-1)
  
  
     # Cross Entropy
     ce = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=labels)
  
     #loss
     loss = tf.reduce_mean(ce)
  
     # Accuracy
     accuracy = tf.reduce_mean(
                              tf.cast(
                                  tf.equal(tf.cast(preds, tf.int32), labels),
                                  tf.float32
                                  )
    #training                         )
    trainop = tf.train.AdamOptimizer().minimize(loss)
  </code></pre>
  
  <p>But In multi-labels how should i take inputs :</p>
  
  <p>so what i tried :</p>
  
  <pre><code>   #weights
      Wo = tf.get_variable('Wo', 
                      shape=[hdim*2, num_labels], 
                      dtype=tf.float32, 
                      initializer=tf.random_uniform_initializer(-0.01, 0.01))
  
  
     #bias
     bo = tf.get_variable('bo', 
                      shape=[num_labels,], 
                      dtype=tf.float32, 
                      initializer=tf.random_uniform_initializer(-0.01, 0.01))
  
  
    #logits
    logits = tf.matmul(tf.concat([fsf.c, fsb.c], axis=-1), Wo) + bo
  
  
     #probability 
     probs  = tf.nn.sigmoid(logits)
  
     #taking max probability
     preds  = tf.round(probs)
  
  
  
  
     # Cross Entropy
     ce = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels)
  
     #loss
     loss = tf.reduce_mean(ce)
  
     # Accuracy
     accuracy = tf.reduce_mean(
                              tf.cast(
                                  tf.equal(tf.cast(preds, tf.int32), labels),
                                  tf.float32
                                  )
    #training                         )
    trainop = tf.train.AdamOptimizer().minimize(loss)
  </code></pre>
  
  <p>Is this right approach , AM i doing right or need correction somewhere ?</p>
  
  <p>If i  passing to tf.nn.sigmoid should i need to pass to  tf.nn.sigmoid_cross_entropy_with_logits or not ??</p>
  
  <p>Thanks You .</p>