title: Confusion about sigmoid derivative's input in backpropagation
body: <p>When using the chain rule to calculate the slope of the cost function relative to the weights at the layer <code>L</code> , the formula becomes:</p>
  <blockquote>
  <p><code>d C0 / d W(L) = ... . d a(L) / d z(L) . ...</code></p>
  </blockquote>
  <p>With :</p>
  <blockquote>
  <p><code>z (L)</code> being the induced local field : <code>z (L) = w1(L) * a1(L-1) + w2(L) * a2(L-1) * ...</code></p>
  <p><code>a (L)</code> beeing the ouput : <code>a (L) = &amp; (z (L)) </code></p>
  <p><code>&amp;</code> being the sigmoid function used as an activation function</p>
  </blockquote>
  <p>Note that <code>L</code> is taken as a layer indicator and not as an index
  <br><br></p>
  <p>Now:<br>
  <code>d a(L) / d z(L) =  &amp;' ( z(L) )</code></p>
  <p>With <code>&amp;'</code> being the derivative of the sigmoid function</p>
  <p><strong>The problem:</strong></p>
  <p>But in this <a href="https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6" rel="nofollow noreferrer">post</a> which is written by James Loy on building a simple neural network from scratch with python,<br> <strong>When doing the backpropagation, he didn't give <code>z (L)</code> as an input to <code>&amp;'</code> to replace <code>d a(L) / d z(L)</code> in the chain rule function. Instead he gave it the <code>output = last activation of the layer (L)</code> as the input the the sigmoid derivative <code>&amp;'</code></strong></p>
  <blockquote>
  <pre><code>def feedforward(self):
          self.layer1 = sigmoid(np.dot(self.input, self.weights1))
          self.output = sigmoid(np.dot(self.layer1, self.weights2))
  
  def backprop(self):
          # application of the chain rule to find derivative of the loss function with respect to weights2 and weights1
          d_weights2 = np.dot(self.layer1.T, (2*(self.y - self.output) * sigmoid_derivative(self.output)))
  </code></pre>
  </blockquote>
  <p>Note that in code above the layer <code>L</code> is the layer <code>2</code> which is the last or output layer.
  And <code>sigmoid_derivative(self.output)</code> this is where the activation of the current layer is given as input to the derivative of the sigmoid function used as an activation function.</p>
  <p><strong>The question:</strong></p>
  <p>Shouldn't we use this <code>sigmoid_derivative(np.dot(self.layer1, self.weights2))</code> instead of this <code>sigmoid_derivative(self.output)</code>?</p>