title: Is backpropagation implemented in TensorFlow when using GradientDescentOptimizer?
body: <p>In the TensorFlow white paper it is stated that gradients are calculated on the backwards path with the chain rule. I visualised the tutorial "<a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_softmax.py" rel="nofollow">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_softmax.py</a>" in Tensorboard and couldn't find out if this is actually happening.
  If there is an edge in the Tensorboard visualisation, can the data flow in both directions? (normally I would expect directed edges)
  <a href="https://i.stack.imgur.com/nFc5Y.png" rel="nofollow"><img src="https://i.stack.imgur.com/nFc5Y.png" alt="Graph visualisation in Tensorboard"></a></p>