title: Why do my loss and accuracy seem to be unrelated?
body: <p>I have a CNN for predicting CIFAR10 classes, which I've adapted from a working solution for the MNIST data-set. </p>
  
  <p>The model is <em>image->conv->pool->conv->pool->FC->dropout->FC->class_prediction</em>, which I assume would do substantially better than a fully-connected net (which I was achieving ~50% with when doing the <a href="http://cs231n.github.io" rel="nofollow noreferrer">CS231 course</a>).</p>
  
  <p>I'm receiving some strange results, where the accuracy and loss don't really seem to be linked to one-another: <a href="https://i.stack.imgur.com/ihXR4.png" rel="nofollow noreferrer">training accuracy and loss graph</a></p>
  
  <p>As shown, the loss keeps dropping, which is a good thing, but it doesn't seem to really affect the accuracy, even on the training set.</p>
  
  <p>Sorry for dumping this all here, but below I've extracted all the code I believe to be relevant. So please let me know if I can add any further information.</p>
  
  <pre><code>with tf.name_scope('Model'):
      # Construct model
      pred = conv_net(x, weights, biases, keep_prob)
  
  with tf.name_scope('Loss'):
      # Define loss function and optimizer
      net = tf.nn.softmax_cross_entropy_with_logits(pred, y_true, name='Softmax_Loss')
      cost = tf.reduce_mean(net)
  
  with tf.name_scope('Adam_Descent'):
      optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam_Descent').minimize(cost)   
  
  # Evaluate the model
  correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(y_true, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
  
  #
  # Code omitted
  #
  
  with tf.Session() as sess:
      sess.run(init)
  
      step = 1
      # Train for a while
      while step * batch_size &lt; training_iters:
  
          batch_x, batch_y = get_rand_train_batch(batch_size, 10000)
  
          # Run optimization        
          _, summary = sess.run([optimizer, merged_summary_op], feed_dict={x: batch_x, 
                                                                           y_true: batch_y, 
                                                                           keep_prob: 0.75})
          if step * index % display_step == 0:
              # Calculate loss and accuracy
              loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x, 
                                                                y_true: batch_y, 
                                                                keep_prob: 1})
              print_blue("Iter " + str(step*batch_size) + ", Minibatch Loss= " + \
                    "{:.6f}".format(loss) + ", Training Accuracy= " + \
                    "{:2.3f}%".format(acc*100))
          summary_writer.add_summary(summary, step * batch_size)
          step += 1
  
      # Calculate test accuracy
      print("\nTesting Accuracy:", \
      sess.run(accuracy, feed_dict={x: X_test[:1000],
                                    y_true: y_test[:1000],
                                    keep_prob: 1.}))
  </code></pre>