<p>The <a href="https://www.iro.umontreal.ca/%7Evincentp/ift3395/lectures/backprop_old.pdf" rel="nofollow noreferrer">backpropagation</a> was created by Rumelhart and Hinton et al and published <a href="https://www.nature.com/articles/323533a0" rel="nofollow noreferrer">on Nature</a> in 1986.</p>
<p>As stated in section <a href="https://www.deeplearningbook.org/contents/mlp.html#page=200" rel="nofollow noreferrer">6.5: Back-Propagation and Other DiﬀerentiationAlgorithms</a>  of the <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">deeplearning book</a> there are two types of approaches for back-propagation gradients through computational graphs: symbol-to-number differentiation and symbol to symbol derivatives. The more relevant one to Tensorflow as stated in this paper: <a href="https://arxiv.org/abs/1610.01178" rel="nofollow noreferrer">A Tour of TensorFlow</a> is the later which can be illustrated using this diagram:</p>
<p><a href="https://i.stack.imgur.com/f0THJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/f0THJ.png" alt="enter image description here" /></a></p>
<p>Source: Section II Part D of <a href="https://arxiv.org/abs/1610.01178" rel="nofollow noreferrer">A Tour of TensorFlow</a></p>
<p>In left side of the the Fig. 7 above, w represents the weights(or Variables) in Tensorflow and x and y are two intermediary operations(or nodes, w, x, y and z are all operations) to get the scalar loss z.</p>
<p>Tensorflow will add a node to each node(if we print the names of variables in a certain checkpoint we can see some additional variables for such nodes and they will be eliminated if we freeze the model to a protocol buffer file for deployment) in the graph for the gradient which can be seen in diagram (b) on the right side: dz/dy, dy/dx, dx/dw.</p>
<p>During the traversal of the back propagation at each node we multiply its gradient with that of the previous one and finally to get a symbolic handle to the overall target derivative dz/dw = dz/dy * dy/dx * dx/dw, which applies exactly the chain rule. Once the gradient is worked out w can update itself with a learning rate.</p>
<p>For more detailed information please read this paper: <a href="https://arxiv.org/pdf/1603.04467.pdf" rel="nofollow noreferrer">TensorFlow:
Large-Scale Machine Learning on Heterogeneous Distributed Systems</a></p>
How do backpropagation works in tensorflow
<p>In tensorflow it seems that the entire backpropagation algorithm is performed by a single running of an optimizer on a certain cost function, which is the output of some MLP or a CNN.</p>

<p>I do not fully understand how tensorflow knows from the cost that it is indeed an output of a certain NN? A cost function can be defined for any model. How should I "tell" it that a certain cost function derives from a NN?</p>