title: How to correctly implement backpropagation for machine learning the MNIST dataset?
body: <p>So, I'm using Michael Nielson's machine learning book as a reference for my code (it is basically identical): <a href="http://neuralnetworksanddeeplearning.com/chap1.html" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com/chap1.html</a></p>
  
  <p>The code in question:</p>
  
  <pre><code>    def backpropagate(self, image, image_value) :
  
  
          # declare two new numpy arrays for the updated weights &amp; biases
          new_biases = [np.zeros(bias.shape) for bias in self.biases]
          new_weights = [np.zeros(weight_matrix.shape) for weight_matrix in self.weights]
  
          # -------- feed forward --------
          # store all the activations in a list
          activations = [image]
  
          # declare empty list that will contain all the z vectors
          zs = []
          for bias, weight in zip(self.biases, self.weights) :
              print(bias.shape)
              print(weight.shape)
              print(image.shape)
              z = np.dot(weight, image) + bias
              zs.append(z)
              activation = sigmoid(z)
              activations.append(activation)
  
          # -------- backward pass --------
          # transpose() returns the numpy array with the rows as columns and columns as rows
          delta = self.cost_derivative(activations[-1], image_value) * sigmoid_prime(zs[-1])
          new_biases[-1] = delta
          new_weights[-1] = np.dot(delta, activations[-2].transpose())
  
          # l = 1 means the last layer of neurons, l = 2 is the second-last, etc.
          # this takes advantage of Python's ability to use negative indices in lists
          for l in range(2, self.num_layers) :
              z = zs[-1]
              sp = sigmoid_prime(z)
              delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
              new_biases[-l] = delta
              new_weights[-l] = np.dot(delta, activations[-l-1].transpose())
          return (new_biases, new_weights)
  </code></pre>
  
  <p>My algorithm can only get to the first round backpropagation before this error occurs:</p>
  
  <pre><code>  File "D:/Programming/Python/DPUDS/DPUDS_Projects/Fall_2017/MNIST/network.py", line 97, in stochastic_gradient_descent
      self.update_mini_batch(mini_batch, learning_rate)
    File "D:/Programming/Python/DPUDS/DPUDS_Projects/Fall_2017/MNIST/network.py", line 117, in update_mini_batch
      delta_biases, delta_weights = self.backpropagate(image, image_value)
    File "D:/Programming/Python/DPUDS/DPUDS_Projects/Fall_2017/MNIST/network.py", line 160, in backpropagate
      z = np.dot(weight, activation) + bias
  ValueError: shapes (30,50000) and (784,1) not aligned: 50000 (dim 1) != 784 (dim 0)
  </code></pre>
  
  <p>I get why it's an error. The number of columns in weights doesn't match the number of rows in the pixel image, so I can't do matrix multiplication. Here's where I'm confused -- there are 30 neurons used in the backpropagation, each with 50,000 images being evaluated. My understanding is that each of the 50,000 should have 784 weights attached, one for each pixel. But when I modify the code accordingly:</p>
  
  <pre><code>        count = 0
          for bias, weight in zip(self.biases, self.weights) :
              print(bias.shape)
              print(weight[count].shape)
              print(image.shape)
              z = np.dot(weight[count], image) + bias
              zs.append(z)
              activation = sigmoid(z)
              activations.append(activation)
              count += 1
  </code></pre>
  
  <p>I still get a similar error:</p>
  
  <pre><code>ValueError: shapes (50000,) and (784,1) not aligned: 50000 (dim 0) != 784 (dim 0)
  </code></pre>
  
  <p>I'm just really confuzzled by all the linear algebra involved and I think I'm just missing something about the structure of the weight matrix. Any help at all would be greatly appreciated.</p>