title: Non-deterministic Gradient Computation
body: <p>I realized that my models end up being different every time I train them, even though I keep the TensorFlow random seed the same.</p>
  
  <p>I verified that:</p>
  
  <ul>
  <li>Initialization is deterministic; the weights are identical before the first update.</li>
  <li>Inputs are deterministic. In fact, various forward computations, including the loss, are identical for the very first batch.</li>
  <li>The gradients for the first batch are different. Concretely, I'm comparing the outputs of <code>tf.gradients(loss, train_variables)</code>. While <code>loss</code> and <code>train_variables</code> have identical values, the gradients are <em>sometimes</em> different for some of the Variables. The differences are quite significant (sometimes the sum-of-absolute-differences for a single variable's gradient is greater than 1).</li>
  </ul>
  
  <p>I conclude that it's the gradient computation that causes the non-determinism.
  I had a look at <a href="https://stackoverflow.com/questions/39938307/determinism-in-tensorflow-gradient-updates">this question</a> and the problem persists when running on a CPU with <code>intra_op_parallelism_thread=1</code> and <code>inter_op_parallelism_thread=1</code>.</p>
  
  <p>How can the backward pass be non-deterministic when the forward pass isn't? How could I debug this further?</p>