title: Is Caffe's reported accuracy reliable?
body: <p>I recently tried to get the confusion matrix for one of my trained models, to see how precise it is.  I downloaded this script and fed my model. To my astonishment, the accuracy calculated by the script, is very different than the one, Caffe reports.</p>
  <p>I have used <a href="https://gist.github.com/axel-angel/b2af7d980eb217a0af07" rel="nofollow noreferrer">this script</a> to calculate the confusion matrix, this however, reports the accuracy as well, <strong>the problem is the accuracy reported by this script is way different that the one reported by Caffe!</strong> For example Caffe reports the accuracy lets say for <code>CIFAR10</code>, as 92.34%, while, when the model is fed to the script to calculate confusion matrix and its accuracy, it results in for example something like 86.5%!</p>
  <p><strong>Which one of these accuracies are the correct one, and can be reported in papers or compared with the results of other papers such as those <a href="http://www.google.com/url?q=http%3A%2F%2Frodrigob.github.io%2Fare_we_there_yet%2Fbuild%2Fclassification_datasets_results.html%2343494641522d313030&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNHlNXPmKuvC0R2w8MsWikjC8O3nuQ" rel="nofollow noreferrer">here</a></strong> ?</p>
  <p>I also saw something weird again, I trained two identical models, with only one difference, that being one used Xavier, and the other used <code>msra</code> for initialization. The first one reports an accuracy of 94.25 and the other reports 94.26 in Caffe. when these models are fed to the script I linked above, for confusion matrix computations. their accuracies were 89.2% and 87.4% respectively!</p>
  <p>Is this normal? what is the cause for this? msra?</p>
  <p>Are the accuracies reported by Caffe true and reliable?</p>
  <p>PS: The accuracy in the script is calculated as (<a href="https://gist.github.com/axel-angel/b2af7d980eb217a0af07" rel="nofollow noreferrer">complete script</a>):</p>
  <pre><code>  for i, image, label in reader:
          image_caffe = image.reshape(1, *image.shape)
          out = net.forward_all(data=np.asarray([ image_caffe ]))
          plabel = int(out['prob'][0].argmax(axis=0))
  
          count += 1
          iscorrect = label == plabel
          correct += (1 if iscorrect else 0)
          matrix[(label, plabel)] += 1
          labels_set.update([label, plabel])
  
          if not iscorrect:
              print(&quot;\rError: i=%s, expected %i but predicted %i&quot; \
                      % (i, label, plabel))
  
          sys.stdout.write(&quot;\rAccuracy: %.1f%%&quot; % (100.*correct/count))
          sys.stdout.flush()
  
      print(&quot;, %i/%i corrects&quot; % (correct, count))
  </code></pre>
  <p>Which imho is OK and correct. the number of correct predictions, divided by the total number of instances in the dataset.</p>