title: Why my gradient descent algorithm for linear regression diverges
body: <p>Gradient descent for linear regression diverges even if I decrease the value for alpha or even delete it!</p>
  <pre><code>def gradientDescent(sqft_living, price, theta, alpha, num_iters):
  m = price.size
  
  # make a copy of theta, to avoid changing the original array, since numpy arrays
  # are passed by reference to functions
  theta = theta.copy()
  
  J_history = [] # Use a python list to save cost in every iteration
  theta_history = []
  for z in range(num_iters):
      term = 0
      term0 = 0        
      for i in range(m):
          h = np.dot(theta, sqft_living[i])
          term += (h - price[i]) 
          term0 += term * sqft_living[i][0]
      temp0 = theta[0] - alpha * (1/m) * term0
  
  
      term = 0
      term1 = 0
      for i in range(m):
          h = np.dot(theta, sqft_living[i])
          term += (h - price[i])
          term1 += term * sqft_living[i][1]
      temp1 = theta[1] - alpha * (1/m) * term1
  
      theta[0] = temp0
      theta[1] = temp1
  
  
      cost = computeCost(sqft_living, price, theta)
      print(cost)
      J_history.append(cost)
  
      theta_history.append(theta)
   
  return theta, J_history
  </code></pre>
  <p>This is the using of the algorithm</p>
  <pre><code># initialize fitting parameters
  theta = np.zeros(2)
  
  # some gradient descent settings
  iterations = 100
  alpha = 0.00001
  
  theta, J_history = gradientDescent(sqft_living, price, theta, alpha, iterations)
  print(J_history)
  </code></pre>
  <p>This is a part of the output :</p>
  <p>cost =  3.8144815615142405e+22</p>
  <p>cost =  8.337226954930875e+33</p>
  <p>cost =  1.8222489338606256e+45</p>
  <p>cost =  3.982848487760674e+56</p>
  <p>cost =  8.705222311669148e+67</p>
  <p>cost =  1.9026808508648173e+79</p>
  <p>cost =  4.158646718757122e+90</p>
  <p>cost =  9.089460549082665e+101</p>
  <p>cost =  1.9866629377457605e+113</p>
  <p>cost =  4.342204476162054e+124</p>
  <p>cost =  9.49065860875058e+135</p>
  <p>cost =  2.074351894810886e+147</p>
  <p>cost =  4.533864256311113e+158</p>