title: Rolling linear regression on large DataFrames
body: <p>I have two huge dataframes <code>df_y</code> and <code>df_x</code>. <br>
  <code>df_y</code> has columns <code>['date','ids','Y']</code>. Basically each <code>'ids'</code> has data for all the <code>'date'</code>.<br>
  <code>df_x</code> has columns <code>['date','X1','X2','X3','X4','X5','X6']</code>. <br>
  <code>df_x</code> has all the <code>date</code> that are in <code>df_y</code>. However some <code>ids</code> might have shorter period, i.e., either starting from a late <code>date</code> or ending 
   at an early <code>date</code>.<br>
  I want to run a rolling linear regression (OLS) <code>Id ~ X1 + X2 + X3 + X4 + X5 + X6 + intercept</code> for each <code>'ids'</code> in <code>df_y</code> with a lookback of 200 days.</p>
  
  <p><strong>Sample dataframes:</strong><br></p>
  
  <pre><code>import string, random, pandas as pd, numpy as np
  ids = [''.join(random.choice(string.ascii_uppercase) for _ in range(3)) for _ in range(200)]
  dates = pd.date_range('2000-01-01', '2017-07-02')
  df_dates = pd.DataFrame({'date':dates, 'joinC':len(dates)*[2]})
  df_ids = pd.DataFrame({'ids':ids, 'joinC':len(ids)*[2]})
  df_values = pd.DataFrame({'Y':np.random.normal(size = 
  len(dates)*len(ids))})
  df_y = df_dates.merge(df_ids, on='joinC', how="outer")
  df_y = df_y[['date', 'ids']].merge(df_values, left_index=True, 
  right_index=True, how="inner")
  df_y = df_y.sort_values(['date', 'ids'], ascending=[True, True])
  df_x = pd.DataFrame({'date':dates, 'X1':np.random.normal(size = len(dates)), 'X2':np.random.normal(size = len(dates)), 'X3':np.random.normal(size = len(dates)), 'X4':np.random.normal(size = len(dates)), 'X5':np.random.normal(size = len(dates)), 'X6':np.random.normal(size = len(dates))})
  </code></pre>
  
  <p><strong>My attempt:</strong></p>
  
  <pre><code>import statsmodels.api as sm
  dates = list(df_y['date'].unique())
  ids = list(df_y['ids'].unique())
  for i in range(200, len(dates) +1):
    for id in ids:
      s_date = dates[i - 200]
      e_date = dates[i - 1]
      Y = df_y[(df_y['date'] &gt;= s_date) &amp; (df_y['date'] &lt;= e_date) &amp; (df_y['ids'] == id)]['Y']
      Y = Y.reset_index()['Y']
      X = df_x[(df_x['date'] &gt;= s_date) &amp; (df_x['date'] &lt;= e_date)]
      X = X.reset_index()[['X1','X2','X3','X4','X5','X6']]
      X = sm.add_constant(X)
      if len(X) &lt;&gt; len(Y):
        continue
      regr = sm.OLS(Y, X).fit()  #Hangs here after 2 years.
      X_pr = X.tail(1)
      Y_hat = regr.predict(X_pr)
      Y.loc[(df_y['date'] == e_date) &amp; (df_y['ids'] == id), 'Y_hat'] = Y_hat.tolist()[0]
  </code></pre>
  
  <p><em>My attempt</em> above seems to be working fine up until the point where it hangs (most likely at fitting step) after running for approx. 2 years. I am inclined to use <code>statsmodels</code> since it supports regularization (planning for future work). However, if using other library makes it faster or more elegant then I am fine with it too. Could someone please help define the fastest solution that doesn't hang midway. Thanks a lot.</p>