title: Different loss functions for backpropagation
body: <p>I came across some different error calculation functions for backpropagation:
  Squared error function from <a href="http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noreferrer">http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/</a></p>
  
  <p><a href="https://i.stack.imgur.com/Nskzd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Nskzd.png" alt="enter image description here"></a></p>
  
  <p>or <a href="https://stackoverflow.com/questions/11627773/back-propagation-error-function#">a nice explanation</a> for the derivation for the BP loss function</p>
  
  <pre><code>Error = Output(i) * (1 - Output(i)) * (Target(i) - Output(i))
  </code></pre>
  
  <p>Now, I'm wondering how many more there are, and what difference in effect it has on training?</p>
  
  <p>Also, since I understand that the second example uses the derivative of the activation function used by the layer, does the first one also does this in a way? And would it be true for any loss function (if there are more)?</p>
  
  <p>Finally, how to know which one to use, and when?</p>