title: I am having trouble training a simple neural network
body: <p>I am learning to make neural network and I have derived the following equations for backpropagation. Is something wrong with this because I can't seem to get the neural network training. I am coding in python and the accuracy I get is around 10 (I get this even when I don't train my network). However, the error is decreasing for each iteration. </p>
  
  <p>Also, I'm currently identifying matrix by size to perform dot product, Hadamard product and transposition, kind of like hit and trial. Is there a better way to do this? </p>
  
  <p>According to my understanding, we try to find the right weights such that the cost is minimized. So, to do that we take partial derivative which gives us a measure of how much a small change in a component will affect the overall cost of the system. So following that I derived this but I don't get why the network is not training. Please help...</p>
  
  <p>Its a 3 layer neural network (input layer, 1 hidden layer, output layer). First I feed the input forward through the neural network, then calculate cost and finally use back-propagation to update the weights. Here, <code>x</code> is the input data, <code>w</code> is the weight (w1 for layer 1 and w2 for layer 2), <code>z(a,w)</code> is the transfer of data from left layer to right layer, <code>g(z)</code> is the activation function (Sigmoid function used), <code>a(z)</code> is the output of the layer, and <code>E(a,y)</code> is the cost function (squared mean error used).</p>
  
  <p>This is my feed forward equation of the neural network:</p>
  
  <p><a href="https://i.stack.imgur.com/dMxZn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dMxZn.png" alt="Feed forward"></a></p>
  
  <p>This is the derivation of back-propagation gradients for weights <code>w1</code>, <code>w1b</code>(bias for layer 1), <code>w2</code> and <code>w2b</code>(bias for layer 2). </p>
  
  <p><a href="https://i.stack.imgur.com/5eGmN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5eGmN.png" alt="derivation of gradients"></a></p>
  
  <p>My python code: 
  (I have used <code>p</code> as a list to transfer weights between functions in sequence <code>w1,w1b,w2,w2b</code>. Also, I added the regularization terms later on so is not included in the derivation. But it follows the same principle of partial derivative. The code was written in Jupyter Notebook and every function has its own block.)</p>
  
  <pre><code>alpha=0.12
  lmda=1
  
  def feedForward(p,x):
      a1=x
      z2=np.dot(a1,p[0])+p[1]
      a2=g(z2)
      z3=np.dot(a2,p[2])+p[3]
      a3=g(z3)    
      return [a1,a2,a3],[z2,z3]
  
  from scipy.special import expit
  def g(z):
      return expit(z)
  
  def gPrime(z):
      s=g(z)
      return np.multiply(s,(1-s))
  
  
  def cost(p,x,y):
      [a,z]=feedForward(p,x)
      h=a[-1]
      lmda_sum=(lmda*0.5/x.shape[0])*(sum(sum(np.square(p[0])))+sum(sum(np.square(p[2])))
      return sum(sum(np.square(y-h)))*0.5/x.shape[0]+lmda_sum
  
  def backpropagate(p,x,y):    
      [a,z]=feedForward(p,x)
      h=a[-1]
      dz=[]
      m=x.shape[0] 
      dz4=np.divide((-(y-h)),m)
      dz3=np.multiply(dz4,gPrime(z[-1]))
      dw2=np.dot(a[-2].T,dz3) + np.multiply(lmda/m,p[2])
      dw2b=np.sum(dz3,0)
  
      dz2=np.multiply(np.dot(dz3,p[2].T),gPrime(z[-2]))
      dw1=np.dot(a[-3].T,dz2) + np.multiply(lmda/m,p[0])
      dw1b=np.sum(dz2,0)
      return [dw1,dw1b,dw2,dw2b]
  
  def costPrime(p,x,y):
      grad=backpropagate(p,x,y)
      p[0] =p[0]-np.multiply(alpha,grad[0])
      p[1] =p[1]-np.multiply(alpha,grad[1])
      p[2] =p[2]-np.multiply(alpha,grad[2])
      p[3] =p[3]-np.multiply(alpha,grad[3])
      return p
  
  def train(p,x,y,iteration):
      E=[]
      for i in range(iteration):
          E.append(cost(p,x,y))
          p=costPrime(p,x,y)
      return p,E
  
  def predict(p,x,y):
      [pred,z]=feedForward(p,x)
      a=pred[-1].argmax(axis=1)
      a=np.reshape(a,(y.shape[0],1))
      b=np.argmax(y,axis=1) # gets index of max element for every row
      return a, np.mean(a==b)*100  
  #     return pred, np.mean(np.sum(pred,0) == np.sum(y,0)) * 100
  
  def randomInitialize(params):
      epsilion = 0.12
      m=len(params)
      for i in range(m):
          s=params[i].shape
          params[i]=np.multiply(np.random.rand(s[0],s[1]),2*epsilion)-epsilion
      return params
  </code></pre>
  
  <p>Below is the main body where I perform everything. It has been broken down into block so as to explain what the block does. The blocks of code combined to form my main body where all of the above functions are called.</p>
  
  <pre><code>%matplotlib inline
  import matplotlib.pyplot as plot
  import random
  import numpy as np
  import scipy.io as io
  </code></pre>
  
  <p>Here I import input and output data from a matrix exported from matlab and store them in <code>X</code> and <code>y</code>.</p>
  
  <pre><code>data=io.loadmat('ex4data1.mat')
  X=data['X']
  y=data['y']
  </code></pre>
  
  <p>Here convert the vector <code>y</code> (which is a vector say mx1) into a matrix (mx10). Each row will be converted such as: if <code>y=3</code> then the 3rd element in the row will be <code>1</code> and rest will be <code>0</code> i.e <code>[0 0 1 0 0 0 0 0 0 0]</code>. For <code>y=0</code>, it is mapped to the 10th index i.e <code>[0 0 0 0 0 0 0 0 0 1]</code>.</p>
  
  <pre><code>y_train=np.zeros((y.shape[0],10))
  for i in range(1,11):
      c=np.zeros((1,10))
      if i==10:
          c[0][0]=1
      else:
          c[0][i]=1 
      y_train[np.where(y==i)[0]] = c
  y=y_train
  </code></pre>
  
  <p>All of the loaded data is used to train the network</p>
  
  <pre><code>xtrain=X
  ytrain=y
  
  w1=np.zeros((400,26))
  w1b=np.zeros((1,26))
  w2=np.zeros((26,10))
  w2b=np.zeros((1,10))
  
  p=[w1,w1b,w2,w2b]
  p=randomInitialize(p)
  [p,E]=train(p,X,y,50)
  plot.plot(E)
  </code></pre>
  
  <p>Randomly select 200 rows from the set <code>X</code> and <code>y</code> then use to as the testing set. I know this should not be done as all of the data has been used for training, but I just wanted to check if the program is even training. If it had successfully trained, it would result a much greater accuracy than 10.</p>
  
  <pre><code>randList=np.random.randint(X.shape[0],size=(1,200))
  xtest=np.reshape(X[randList],(200,-1))
  ytest=np.reshape(y[randList],(200,10))
  
  [pred,acc]=predict(p,xtest,ytest)
  print acc
  </code></pre>
  
  <p>This is the cost graph I get after plotting <code>E</code> which is returned by the <code>train(p,X,y,50)</code>.</p>
  
  <p><a href="https://i.stack.imgur.com/Y9Rjn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Y9Rjn.png" alt="enter image description here"></a></p>