title: tensorflow mean_iou - how to get consistent results
body: <p>I am stumped by the tensorflow metric mean_iou().  My attempt to use leads to inconsistent results on the same data, and I can't find a useful example anywhere.</p>
  
  <p>this blog post talks about the issue...  but I can't solve my problems with the ideas here.  <a href="http://ronny.rest/blog/post_2017_09_11_tf_metrics/" rel="noreferrer">http://ronny.rest/blog/post_2017_09_11_tf_metrics/</a></p>
  
  <p>End goal is to have a <code>validation_func(y_true, y_pred)</code> to pass as arg 'metric' to a Keras Model.compile() method.</p>
  
  <p><strong>end goal</strong></p>
  
  <p>something like...</p>
  
  <pre><code>def my_iou(y_true, y_pred):
    num_classes = 2
    score, update_op = tf.metrics.mean_iou(tf.argmax(y_true, axis=3), tf.argmax(y_pred, axis=3), num_classes)
    K.get_session().run(tf.local_variables_initializer())
    # K.get_session().run(tf.global_variables_initializer())
    with tf.control_dependencies([update_op]):
      final_score = tf.identity(score)
    return final_score
  </code></pre>
  
  <p><strong>dummy data</strong></p>
  
  <ul>
  <li>dimensions will be: (batch, height, width, num_classes)</li>
  <li>class 1 - foreground object; what I'm trying to segment out</li>
  <li>class 0 - background</li>
  </ul>
  
  <p>what I really care about is IOU on class 1.  in this case that should be 7/9=0.778.  what I expect this to spit out though is an average of iou on each class...  </p>
  
  <ul>
  <li>iou class 0: 0/2=0</li>
  <li>iou class 1: 7/9=0.778</li>
  <li>overall: avg(iou_0, iou_1)=0.3889</li>
  </ul>
  
  <p>data</p>
  
  <pre><code>from keras import backend as K
  import tensorflow as tf
  import numpy as np
  
  y_true = np.zeros(18, dtype='int32').reshape((1,3,3,2))
  y_true[:,:,:,1] = 1
  y_true[:,0,0,1] = 0
  y_true[:,0,0,0] = 1
  
  y_pred = np.zeros(18, dtype='float32').reshape((1,3,3,2))
  y_pred[:,:,:,1] = 1.0
  y_pred[:,-1,-1,1] = 0.1
  y_pred[:,-1,-1,0] = 0.9
  </code></pre>
  
  <p>If I simulate with something like the following, I get the expected 0.3889 most of the time...  but i also get occasional 3.5 or 0.0, and I can't figure out why?</p>
  
  <pre><code>results = []
  for i in range(20):
    with tf.Session() as sess:
      y_t = tf.convert_to_tensor(y_true)
      y_p = tf.convert_to_tensor(y_pred)
      results.append(my_iou(y_t, y_p).eval())
  </code></pre>
  
  <p>a recent run</p>
  
  <pre><code>[0.3888889,
   0.3888889,
   0.3888889,
   0.3888889,
   3.5,
   0.3888889,
   0.3888889,
   0.3888889,
   0.3888889,
   0.3888889,
   0.3888889,
   0.3888889,
   0.3888889,
   3.5,
   0.3888889,
   0.3888889,
   0.3888889,
   3.5,
   0.3888889,
   0.3888889]
  </code></pre>
  
  <p>any help?</p>
  
  <p><strong>EDIT</strong> </p>
  
  <p>for now I'm using something hard coded specifically for binary classification, but I don't love it.  Not generalizable, probably too simple, and maybe outright wrong.  (also of note...  in this code, class 0 is the foreground, unlike my example setup where class 1 was the foreground)</p>
  
  <pre><code>def my_iou_binary(self, y_true, y_pred):
    yt0 = y_true[:,:,:,0]
    yp0 = K.cast(y_pred[:,:,:,0] &gt; 0.5, 'float32')
  
    inter = tf.count_nonzero(tf.logical_and(tf.equal(yt0, 1), tf.equal(yp0, 1)))
    union = tf.count_nonzero(tf.add(yt0, yp0))
    iou = tf.where(tf.equal(union, 0), 1., tf.cast(inter/union, 'float32'))
  
    return iou
  </code></pre>