title: Deep neural network gives a constant value as output
body: <p>I have implemented a neural network from scratch. I have taken random values as both input and output. When I use a single layer( with sigmoid activation) the network performs well. But when I increase the number of layers( No matter which activation I use), The network outputs a constant value.</p>
  <p>I have attached the link to the full code and output for single layer and and a network with 2 hidden layers.</p>
  <p>HyperParameter (relu,sigmoid,derRelu and derSigmoid are functions)</p>
  <pre><code>alpha= 0.01 #learning rate
  m=10 #number of examples in training set
  numberOfLayers=2 #including input and output layer
  activations=[None,relu,sigmoid]
  derActivations=[None,derRelu,derSigmoid]
  neurons=[10,5,1]
  numberOfIterations=1000
  </code></pre>
  <p>Initialization of parameters:</p>
  <pre><code>parameters=[]
  parameters.append({})
  
  for i in range(1,numberOfLayers): 
      param={}
      param[&quot;W&quot;]  =np.random.randn(neurons[i],neurons[i-1])*0.01
      param[&quot;b&quot;]  =np.zeros((neurons[i],1))
      parameters.append(param)
  </code></pre>
  <p>These are some of the functions</p>
  <pre><code>def calculateZ(A,W,b):
  Z=np.dot(W,A)+b
  return Z
  
  def derL(Y,A):
  derL=np.true_divide(1-Y,1-A)-np.true_divide(Y,A)
  return derL
  
  def derW(dZ,A):
  dW=np.dot(dZ,np.transpose(A))/m;
  return dW
  
  def derB(dZ):
  db=np.sum(dZ,axis=1,keepdims=True)/m;
  return db
  
  def derZ(dW,dZ,Z,derActi):
  dZNew=np.multiply(np.dot(np.transpose(dW),dZ),derActi(Z))
  return dZNew
  
  def forwardPropagation(layer):
      W=parameters[layer][&quot;W&quot;]    #current layer
      b=parameters[layer][&quot;b&quot;]    #current layer
      A=parameters[layer-1][&quot;A&quot;]  #previous layer
  
      Z=calculateZ(A,W,b)
      parameters[layer][&quot;Z&quot;]=Z    #current layer
  
  
      A=activations[layer](Z)
      parameters[layer][&quot;A&quot;]=A
  
  def backwardPropagation(layer):
      Z=parameters[layer][&quot;Z&quot;]    #current layer
  
      if(layer==numberOfLayers-1):
          A=parameters[layer][&quot;A&quot;]
          dL=derL(Y,A)
          dZ=np.multiply(dL,derActivations[layer](Z))
      else:
          dZ=parameters[layer+1][&quot;dZ&quot;]  #next layer
          dW=parameters[layer+1][&quot;dW&quot;]  #next layer    
          dZ=derZ(dW,dZ,Z,derActivations[layer])  #current layer
  
      parameters[layer][&quot;dZ&quot;]=dZ
      A=parameters[layer-1][&quot;A&quot;]  #previous layer
  
      dW=derW(dZ,A)               #current layer
      parameters[layer][&quot;dW&quot;]=dW
      db=derB(dZ)               #current layer
      parameters[layer][&quot;db&quot;]=db
  </code></pre>
  <p>Training</p>
  <pre><code>Y=np.random.randn(1,m)
  Y=np.where(Y&lt;0,0,1)
  
  # genrating random input to test the code
  parameters[0][&quot;A&quot;]=np.random.randn(neurons[0],m)
  costs=[]
  
  
  for i in range(numberOfIterations):
      for layer in range(1,numberOfLayers):
          forwardPropagation(layer)
  
      
      A=parameters[numberOfLayers-1][&quot;A&quot;]
      l=calculateLoss(Y,A)
      costs.append(l[0][0])
  
      for layer in range(numberOfLayers-1,0,-1):
          backwardPropagation(layer)
  
      for layer in range(1,numberOfLayers): 
          updateParameters(layer)
  </code></pre>
  <p><a href="https://github.com/hemant239/DeepNeuralNetwork" rel="nofollow noreferrer">Code</a></p>
  <p><a href="https://i.stack.imgur.com/0r8Br.png" rel="nofollow noreferrer">output with 0 hidden layer</a></p>
  <p><a href="https://i.stack.imgur.com/7RYTH.png" rel="nofollow noreferrer">output with 2 hidden layers</a></p>
  <p>I am unable to find the problem.</p>