title: Cumulative OLS with Python Pandas
body: <p>I am using Pandas 0.8.1, and at the moment I can't change the version. If a newer version will help the problem below, please note it in a comment rather than an answer. Also, this is for a research replication project, so even though re-running a regression after appending only one new data point might be silly (if the data set is large), I still have to do it. Thanks!</p>
  
  <p>In Pandas, there is a <code>rolling</code> option for the <code>window_type</code> argument to <code>pandas.ols</code> but it seems implicit that this requires some choice of a window size or use of the whole data sample as default. I'm looking to instead use all the data in a cumulative fashion.</p>
  
  <p>I am trying to run a regression on a <code>pandas.DataFrame</code> that is sorted by date. For each index <code>i</code>, I want to run a regression using the data available from the minimum date up through the date at index <code>i</code>. So the window effectively grows by one on every iteration, all data is cumulatively used from the earliest observation, and no data is ever dropped out of the window.</p>
  
  <p>I have written a function (below) that works with <code>apply</code> to perform this, but it is unacceptably slow. Instead, is there a way to use <code>pandas.ols</code> to directly perform this sort of cumulative regression?</p>
  
  <p>Here are some more specifics about my data. I have a <code>pandas.DataFrame</code> containing a column of identifier, a column of dates, a column of left-hand-side values, and a column of right-hand-side values. I want to use <code>groupby</code> to group based on the identifier, and then perform a cumulative regression for every time period consisting of the left-hand and right-hand-side variables.</p>
  
  <p>Here is the function I am able to use with <code>apply</code> on the identifier-grouped object:</p>
  
  <pre><code>def cumulative_ols(
                     data_frame, 
                     lhs_column, 
                     rhs_column, 
                     date_column,
                     min_obs=60
                    ):
  
      beta_dict = {}
      for dt in data_frame[date_column].unique():
          cur_df = data_frame[data_frame[date_column] &lt;= dt]
          obs_count = cur_df[lhs_column].notnull().sum()
  
          if min_obs &lt;= obs_count:
              beta = pandas.ols(
                                y=cur_df[lhs_column],
                                x=cur_df[rhs_column],
                               ).beta.ix['x']
              ###
          else:
              beta = np.NaN
          ###
          beta_dict[dt] = beta
      ###
  
      beta_df = pandas.DataFrame(pandas.Series(beta_dict, name="FactorBeta"))
      beta_df.index.name = date_column
      return beta_df
  </code></pre>