title: What does opt.apply_gradients() do in TensorFlow?
body: <p>The documentation is not quite clear about this. I suppose the gradients one can obtain by <code>opt.compute_gradients(E, [v])</code> contain the <code>∂E/∂x = g(x)</code> for each element <code>x</code> of the tensor that <code>v</code> stores. Does <code>opt.apply_gradients(grads_and_vars)</code> essentially execute <code>x ← -η·g(x)</code>, where <code>η</code> is the learning rate? That would imply that if I want to add a positive additive change <code>p</code> to the variable, I would need to need to change <code>g(x) ← g(x) - (1/η)p</code>, e.g. like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>opt = tf.train.GradientDescentOptimizer(learning_rate=l)
  grads_and_vars = opt.compute_gradients(loss, var_list)
  
  for l, gv in enumerate(grads_and_vars):
      grads_and_vars[l] = (gv[0] - (1/l) * p, gv[1])
  
  train_op = opt.apply_gradients(grads_and_vars)
  </code></pre>
  
  <p>Is there a better way to do this?</p>