title: Why is binary_crossentropy more accurate than categorical_crossentropy for multiclass classification in Keras?
body: <p>I'm learning how to create convolutional neural networks using Keras. I'm trying to get a high accuracy for the MNIST dataset.</p>
  
  <p>Apparently <code>categorical_crossentropy</code> is for more than 2 classes and <code>binary_crossentropy</code> is for 2 classes. Since there are 10 digits, I should be using <code>categorical_crossentropy</code>. However, after training and testing dozens of models, <code>binary_crossentropy</code> consistently outperforms <code>categorical_crossentropy</code> significantly.</p>
  
  <p>On Kaggle, I got 99+% accuracy using <code>binary_crossentropy</code> and 10 epochs. Meanwhile, I can't get above 97% using <code>categorical_crossentropy</code>, even using 30 epochs (which isn't much, but I don't have a GPU, so training takes forever).</p>
  
  <p>Here's what my model looks like now:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>model = Sequential()
  model.add(Convolution2D(100, 5, 5, border_mode='valid', input_shape=(28, 28, 1), init='glorot_uniform', activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Convolution2D(100, 3, 3, init='glorot_uniform', activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.3))
  model.add(Flatten())
  model.add(Dense(100, init='glorot_uniform', activation='relu'))
  model.add(Dropout(0.3))
  model.add(Dense(100, init='glorot_uniform', activation='relu'))
  model.add(Dropout(0.3))
  model.add(Dense(10, init='glorot_uniform', activation='softmax'))
  model.compile(loss='binary_crossentropy', optimizer='adamax', metrics=['accuracy'])
  </code></pre>