title: Why does the cost in my Linear Regression function increase if I increase the number of iterations?
body: <p>I have been trying to implement a linear regression model with gradient descent (from scratch). In doing so, I have reached a point where increasing the number of iterations my code runs for leads to an obviously worse solution with a supposedly lower cost. Kindly help me.</p>
  <p>Here is the data-set I am using to train the model: <a href="https://www.kaggle.com/datasets/andonians/random-linear-regression" rel="nofollow noreferrer">https://www.kaggle.com/datasets/andonians/random-linear-regression</a> (Train.csv)</p>
  <p>Importing Python Libraries:</p>
  <pre class="lang-py prettyprint-override"><code>import numpy as np # linear algebra
  import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
  import seaborn as sns #data visualisation (seaborn)
  import matplotlib.pyplot as plt
  
  # Input data files are available in the read-only &quot;../input/&quot; directory
  # For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory
  
  import os
  for dirname, _, filenames in os.walk('/kaggle/input'):
      for filename in filenames:
          print(os.path.join(dirname, filename))
  </code></pre>
  <p>I am importing the data as follows:</p>
  <pre class="lang-py prettyprint-override"><code>data = pd.read_csv(&quot;../input/random-linear-regression/train.csv&quot;)
  data.head()
  </code></pre>
  <p>Here is the Linear Regression class:</p>
  <pre class="lang-py prettyprint-override"><code>class LinearRegression() :
      def __init__ (self, iterations = 100, learningrate=0.0001):
          self.iterations = iterations
          self.learningrate = learningrate
      
      def gradientDescent(self,x,y,threshold=1e-9):
          
          x = (x - x.mean())/x.std() #Not sure if this is needed or not
          y = (y-y.mean())/y.std() #Not sure if needed or not
          
          #adding col of 1s to accommodate bias (theta0)        
          x = np.c_[np.ones(x.shape[0]),x] # m x n matrix
  
          
          #Cleaning data to get rid of NAN and replacing with 0 
          y = np.nan_to_num(y)
          
          #m = number of samples
          #n = number of features
          m = x.shape[0]
          n = x.shape[1]
                  
          #weights
          theta = np.ones(n) # n x 1 vector
          
          
          past_thetas = [0]
          past_costs = []
         
          for i in range(self.iterations):
              #hypothesis
              h = np.dot(x,theta) # m x 1 vector
              
              error = np.subtract(h,y) # m x 1   
              errorsq = np.square(error)
              
              cost = 1/(2 * m) * np.sum(errorsq)
              past_costs.append(cost)
  
              delta = (1/m) * np.dot(x.T,error)
              theta = theta - (self.learningrate*delta)
              past_thetas.append(theta)
              
          print(&quot;Weights:&quot;, theta)
          return past_thetas,past_costs
  </code></pre>
  <p>Here is the main function:</p>
  <pre class="lang-py prettyprint-override"><code>def main():
      past_thetas = []
      past_costs = []    
      regressor = LinearRegression()
      past_thetas, past_costs = regressor.gradientDescent(x=data['x'],y=data['y'])
      print(past_costs[-1])
      
      plt.figure(figsize=(10,10))
      sns.scatterplot(data=data, x='x', y='y')
      x = np.linspace(0,100,5)
      y = past_thetas[-1][0] + (past_thetas[-1][1]*x)
      plt.plot(x,y,'b')
      sns.scatterplot(data=data, x='x', y='y')
      
      plt.figure (figsize = (5,5)) 
      plt.plot(past_costs)
  
  if __name__ == &quot;__main__&quot;:
      main()
  </code></pre>
  <p>The problem:
  If I run this with Learning Rate = 0.0001 &amp; Iterations = 100, I get the following results:
  <img src="https://i.stack.imgur.com/59h2P.png" alt="Model plot for 100 iterations" /></p>
  <p>However, if I keep the learning rate constant but change the iterations to 1000 or 10000, the line keeps drifting away from the points and while doing so, it computes a cost lower than an obviously more accurate fit. Here is the result with LR = 0.0001 &amp; iterations = 10000:
  <img src="https://i.stack.imgur.com/vtSqm.png" alt="Model plot for 10000 iterations" /></p>
  <p>Here are the cost function plots in both cases as well:
  <img src="https://i.stack.imgur.com/vZ2l7.png" alt="Cost function plot for # of iterations = 100" />,
  <img src="https://i.stack.imgur.com/LEk8s.png" alt="Cost function plot for # of iterations = 10000" /></p>
  <p>How do I fix this? As per my understanding, more iterations should only make the convergence slower but it should not overshoot the local minimum. The code gives similar results with an even lower learning rate due to which I am fairly sure that the learning rate is not an issue (although please correct me if I am wrong about that)</p>