title: How does one debug NaN values in TensorFlow?
body: <p>I was running TensorFlow and I happen to have something yielding a NaN. I'd like to know what it is but I do not know how to do this. The main issue is that in a "normal" procedural program I would just write a print statement just before the operation is executed. The issue with TensorFlow is that I cannot do that because I first declare (or define) the graph, so adding print statements to the graph definition does not help. Are there any rules, advice, heuristics, anything to track down what might be causing the NaN?</p>
  
  <hr>
  
  <p>In this case I know more precisely what line to look at because I have the following:</p>
  
  <pre><code>Delta_tilde = 2.0*tf.matmul(x,W) - tf.add(WW, XX) #note this quantity should always be positive because its pair-wise euclidian distance
  Z = tf.sqrt(Delta_tilde)
  Z = Transform(Z) # potentially some transform, currently I have it to return Z for debugging (the identity)
  Z = tf.pow(Z, 2.0)
  A = tf.exp(Z) 
  </code></pre>
  
  <p>when this line is present I have it that it returns NaN as declared by my summary writers. Why is this? Is there a way to at least explore what value Z has after its being square rooted?</p>
  
  <hr>
  
  <p>For the specific example I posted, I tried <code>tf.Print(0,Z)</code> but with no success it printed nothing. As in:</p>
  
  <pre><code>Delta_tilde = 2.0*tf.matmul(x,W) - tf.add(WW, XX) #note this quantity should always be positive because its pair-wise euclidian distance
  Z = tf.sqrt(Delta_tilde)
  tf.Print(0,[Z]) # &lt;-------- TF PRINT STATMENT
  Z = Transform(Z) # potentially some transform, currently I have it to return Z for debugging (the identity)
  Z = tf.pow(Z, 2.0)
  A = tf.exp(Z) 
  </code></pre>
  
  <p>I actually don't understand what <code>tf.Print</code> is suppose to do. Why does it need two arguments? If I want to print 1 tensor why would I need to pass 2? Seems bizarre to me.</p>
  
  <hr>
  
  <p>I was looking at the function <a href="https://www.tensorflow.org/versions/r0.9/api_docs/python/control_flow_ops.html#add_check_numerics_ops" rel="noreferrer">tf.add_check_numerics_ops()</a> but it doesn't say how to use it (plus the docs seem to not be super helpful). Does anyone know how to use this?</p>
  
  <hr>
  
  <p>Since I've had comments addressing the data might be bad, I am using standard MNIST. However, I am computing a quantity that is positive (pair-wise eucledian distance) and then square rooting it. Thus, I wouldn't see how the data specifically would be an issue.</p>