title: Implement Cost Function of Neural Network (Week #5 Coursera) using Python
body: <p>Based on the Coursera Course for Machine Learning, I'm trying to implement the cost function for a neural network in python. There is a <a href="https://stackoverflow.com/questions/21441457/neural-network-cost-function-in-matlab">question</a> similar to this one -- with an accepted answer -- but the code in that answers is written in octave. Not to be lazy, I have tried to adapt the relevant concepts of the answer to my case, and as far as I can tell, I'm implementing the function correctly. The cost I output differs from the expected cost, however, so I'm doing something wrong.</p>
  
  <p>Here's a small reproducible example:</p>
  
  <p>The following link leads to an <code>.npz</code> file which can be loaded (as below) to obtain relevant data. Rename the file <code>"arrays.npz"</code> please, if you use it.</p>
  
  <p><a href="http://www.filedropper.com/arrays_1" rel="nofollow noreferrer">http://www.filedropper.com/arrays_1</a></p>
  
  <pre><code>if __name__ == "__main__":
  
  with np.load("arrays.npz") as data:
  
      thrLayer = data['thrLayer'] # The final layer post activation; you 
      # can derive this final layer, if verification needed, using weights below
  
      thetaO = data['thetaO'] # The weight array between layers 1 and 2
      thetaT = data['thetaT'] # The weight array between layers 2 and 3
  
      Ynew = data['Ynew'] # The output array with a 1 in position i and 0s elsewhere
  
      #class i is the class that the data described by X[i,:] belongs to
  
      X = data['X'] #Raw data with 1s appended to the first column
      Y = data['Y'] #One dimensional column vector; entry i contains the class of entry i
  
  
  
  import numpy as np
  
  m = len(thrLayer)
  k = thrLayer.shape[1]
  cost = 0
  
  for i in range(m):
      for j in range(k):
          cost += -Ynew[i,j]*np.log(thrLayer[i,j]) - (1 - Ynew[i,j])*np.log(1 - thrLayer[i,j])
  print(cost)
  cost /= m
  
  '''
  Regularized Cost Component
  '''
  
  regCost = 0
  
  for i in range(len(thetaO)):
      for j in range(1,len(thetaO[0])):
          regCost += thetaO[i,j]**2
  
  for i in range(len(thetaT)):
      for j in range(1,len(thetaT[0])):
          regCost += thetaT[i,j]**2
  
  regCost *= lam/(2*m) 
  
  
  print(cost)
  print(regCost)
  </code></pre>
  
  <p>In actuality, <code>cost</code> should be 0.287629 and <code>cost + newCost</code> should be 0.383770.</p>
  
  <p>This is the cost function posted in the question above, for reference:</p>
  
  <hr>
  
  <p><a href="https://i.stack.imgur.com/WvX7X.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WvX7X.png" alt="enter image description here"></a></p>