<p>The usual representation is</p>

<pre><code>    PREDICTED
 [[41  2  0  0  0  0]
T [ 1 11  4  1  0  0]
R [ 0  1  12 0  0  0]
U [ 0  0  0 22  1  0]
E [ 0  0  0  0  7  0]
  [ 0  0  0  0  0 20]]
</code></pre>

<p>As pointed out by M. Rath (+1), this is also what Tensorflow does. This means for 41 samples you correctly predicted class 0. For 2 samples, you predicted class 1, but it actually was class 0.</p>

<p>Please note that you can also manipulate the order for visualizations. So instead of</p>

<pre><code>class 0, class 1, class 2
</code></pre>

<p>you could have (for both, prediction and true value) the order</p>

<pre><code>class 0, class 2, class 1
</code></pre>

<p>This contains the same information, but a visualization might convey a different story. See my masters thesis <a href="https://arxiv.org/pdf/1707.09725.pdf#page=48" rel="nofollow noreferrer">Analysis and Optimization of
Convolutional Neural Network Architectures</a> page 48 (Confusion Matrix Ordering), especially figure 5.12 and 5.13.</p>

<p>An implementation can be found in the tool <a href="https://github.com/MartinThoma/clana" rel="nofollow noreferrer"><code>clana</code></a></p>
Confusion matrix order in tensorflow
<p>I have 6 classes and I used tf-slim in Tensorflow to obtained the confusion matrix such as</p>

<pre><code>[[41  2  0  0  0  0]
 [ 1 11  4  1  0  0]
 [ 0  1  12  0  0  0]
 [ 0  0  0 22  1  0]
 [ 0  0  0  0  7  0]
 [ 0  0  0  0  0 20]]
</code></pre>

<p>My question is that what is confusion matrix order of the above table? Is it right if I said that the columns represent the prediction label, while the rows represent the true label? Some reference said on opposite side.</p>