title: Python statsmodels: memory error
body: <p>I'm trying to do vector autoregression. The time series has 526 examples (days) with 972196 columns (each entry in the matrix is a single float value).</p>
  
  <p>I'm running into memory errors using Python's <code>VAR</code> module:</p>
  
  <pre><code>user:~/path/autoregression$ python Mar12_autoregression.py
  /user/pkgs/anaconda2/lib/python2.7/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
    from pandas.core import datetools
  /user/pkgs/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/var_model.py:461: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
  To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`. 
    params = np.linalg.lstsq(z, y_sample)[0]
  Traceback (most recent call last):
    File "Mar12_autoregression.py", line 124, in &lt;module&gt;
      var_autoregression(adj_orig, num_nodes)
    File "Mar12_autoregression.py", line 95, in var_autoregression
      lag_7_results = model.fit(7)
    File "/user/pkgs/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/var_model.py", line 438, in fit
      return self._estimate_var(lags, trend=trend)
    File "/user/pkgs/anaconda2/lib/python2.7/site-packages/statsmodels/tsa/vector_ar/var_model.py", line 461, in _estimate_var
      params = np.linalg.lstsq(z, y_sample)[0]
    File "/user/pkgs/anaconda2/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 2004, in lstsq
      bstar = zeros((ldb, n_rhs), t)
  MemoryError 
  </code></pre>
  
  <p>My code is:</p>
  
  <pre><code>import numpy as np
  import sys
  import pandas as pd
  import statsmodels.api as sm
  from scipy.sparse import csr_matrix
  from statsmodels.tsa.api import VAR, DynamicVAR
  
  def var_autoregression(adj_orig, num_nodes, seq_length=7):
      #adj_orig is a dict object with int keys and sparse_csr_matrix values, 
      #unless there is an int key with an empty list value.
  
      intermediate_build = []
      for i in range(len(adj_orig.keys())):
              #Build a 526x(986*986) matrix and convert to a pandas dataframe. Remove NAs.
              curr = adj_orig[adj_orig.keys()[i]]
              if isinstance(curr, list):
                      makevec = 100*np.ones((num_nodes*num_nodes)) #986*986 
              else:   
                      makevec = curr.toarray().flatten() #986*986
              intermediate_build.append(makevec)
  
      #Convert the list with 526 elements, each with size 986*986, to a pandas dataframe with 526 rows and 986*986 columns.
  
      i_build = np.asarray(intermediate_build)
      #print i_build.shape, 'here' #(526, 986*986) as it should be.
  
      col_list = [str(i) for i in range(num_nodes*num_nodes)]
      pd1 = pd.DataFrame(i_build, columns=col_list)
  
      #Need to insert dates into the first column
      pd1.index = pd.date_range("2016-01-01", "2017-06-09")
      pd2 = pd1[pd1['0'] != 100]
  
      model = VAR(pd2)
  
      lag_7_results = model.fit(7)
      print lay_7_results.summary()
  
      return 0
  </code></pre>
  
  <p>Any suggestions on how I can do vector autoregression on big data?</p>
  
  <p><strong>EDIT (responding to a comment):</strong></p>
  
  <p>Each row of 972196 is a flattened adjacency matrix. This is not images; it's graphs. I'm tracking how the adjacency matrix evolves over time. </p>
  
  <p>Small example: say there are 3 people in the graph, and they exchange messages over 526 days. For each day, if there is communication between person i and person j, I record a '1' in the adjacency matrix in row i, column j.</p>
  
  <p>If on the first day, person 0 sends a message to person 1, and person 2 sends a message to person 1, and there are no other messages sent, then the adjacency matrix looks like this:</p>
  
  <pre><code>0 1 0
  0 0 0
  0 1 0
  </code></pre>
  
  <p>If on the second day, person 1 sends a message to person 0 and person 1 sends a message to person 2 and there are no other messages sent, then the adjacency matrix looks like this:</p>
  
  <pre><code>0 0 0
  1 0 1
  0 0 0 
  </code></pre>
  
  <p>And so on, for 526 days. What I'm doing is flattening each adjacency matrix to a vector to be able to feed it in to an autoregression routine. That is, day 1 would look like this:</p>
  
  <pre><code>0, 1, 0, 0, 0, 0, 0, 1, 0
  </code></pre>
  
  <p>and day 2 would look like this:</p>
  
  <pre><code>0, 0, 0, 1, 0, 1, 0, 0, 0
  </code></pre>
  
  <p>I then preprocess these binary vectors in a particular way to make the values floats instead of integers. Suppose for the sake of argument that day 1 becomes:</p>
  
  <pre><code>0, 0.1111, 0, 0, 0, 0, 0, 0.1111, 0
  </code></pre>
  
  <p>and day 2 becomes:</p>
  
  <pre><code>0, 0, 0, 0.2222, 0, 0.2222, 0, 0, 0
  </code></pre>
  
  <p>I'm trying to use past instances of the adjacency matrix (flattened, preprocessed vectors) to predict the next instance.</p>