title: Theano gradients failing over scan operation
body: <p>Implementing deep networks in theano gives me precise control over my layers from the ground up. I know it's no longer supported but it appears to me there is still active use (from me at least :->). Anyway, I noticed some behavior related to gradients calculation through a scan operation.</p>
  
  <p>I have the following inner function (part of a RNN inner iteration step, I can provide if nec.):</p>
  
  <pre><code>    def iter_step(x_step, h):
          ...
  
          return hidden, obj, est
  </code></pre>
  
  <p>I have a set of parameters <code>h_params</code>, <code>o_params</code>, <code>e_params</code> that I use to compute gradients:</p>
  
  <pre><code>h, o, e = iter_step(x_step, h)
  hidden_grads = T.grad(T.sum(h), h_params)
  obj_grads = T.grad(T.sum(o), o_params)
  est_grads = T.grad(T.sum(est), e_params)
  </code></pre>
  
  <p>everything is fine. I add a scan operation</p>
  
  <pre><code>[h_n, obj_n, x_n], _ = theano.scan(
      fn=iter_step,
      sequences=[x],
      outputs_info=[T.as_tensor_variable(np.zeros(model.h_shape), model.h.dtype),
                    None,
                    None],
      )
  </code></pre>
  
  <p>Evaluation is fine, but computing gradients on the same parameters is now a problem:</p>
  
  <pre><code># Works
  h_n0 = theano.function([], h_n)()
  # Fails
  h_n_grads = T.grad(T.sum(h_n), h_params)
  
  ---------------------------------------------------------------------------
  NullTypeGradError                         Traceback (most recent call last)
  &lt;ipython-input-32-0d7c8a65d265&gt; in &lt;module&gt;
  ----&gt; 1 h_n_grads = T.grad(T.sum(h_n), h_params)
  
  /usr/local/lib/python3.6/dist-packages/theano/gradient.py in grad(cost, wrt, consider_constant, disconnected_inputs, add_names, known_grads, return_disconnected, null_gradients)
      609             if null_gradients == 'raise':
      610                 raise NullTypeGradError("tensor.grad encountered a NaN. " +
  --&gt; 611                                         rval[i].type.why_null)
      612             else:
      613                 assert null_gradients == 'return'
  
  NullTypeGradError: tensor.grad encountered a NaN. This variable is Null because the grad method for input 4 (Subtensor{int64}.0) of the for{cpu,scan_fn} op is mathematically undefined. Depends on a shared variable
  </code></pre>
  
  <p>Why is this? I have not been able to debug yet - graph is not disconnected, unrolling scan by hand gives good gradients. The gradient calculations should carry through the scan operation. If gradients on h (1st output of iter_step) can be calculated, why not on the analogous output of the scan?</p>