title: Scikit-learn Random Forest taking up too much memory
body: <h3>Problem</h3>
  
  <p>I have a dataset with 900,000 rows and 8 columns. 6 of the columns are integers and the other two are floats. When trying to fit about 1/4 of the dataset (200,000) the code runs fine and takes under 30 seconds. When I try to run 400,000 rows or greater My computer permanently freezes because the python.exe process takes up over 5GB of RAM.</p>
  
  <h3>Attempts</h3>
  
  <p>First thing I tried was setting the warm_state parameter to True and then going through the data 50,000 rows at a time</p>
  
  <pre><code>n = 0
  i = 50,000  
  clf = sk.RandomForestClassifier(oob_score = True,n_jobs=-1, n_estimators = n, warm_start=True)
  While i&lt;= 850,000:
      clf.fit(X.ix[n:i],Y.ix[n:i])
      n += 50,000
      i += 50,000
  </code></pre>
  
  <p>This didn't solve anything, I ran into the same issue.</p>
  
  <p>Next thing I tried is finding if there was a part of the data that was taking much more memory to process. I recorded the memory increase in the python.exe process and the time it took for the process to complete, if it did complete.</p>
  
  <pre><code>n = 50
  clf = sk.RandomForestClassifier(oob_score = True,n_jobs=-1, n_estimators = n, warm_start=True)
  Z = X[['DayOfWeek','PdDistrict','Year','Day','Month']] # takes 15s and additional ~600mb RAM (800 total)
  Z = X[['X','Address','Y']] # takes 24.8s and additional 1.1GB RAM (1389mb total)
  Z = X # never finishes peaks at 5.2GB
  %time clf.fit(Z.ix[0:400000],Y.ix[0:400000])
  </code></pre>
  
  <p>While some data does take longer to process than others none of them can account for 5 Gb of memory being taken.</p>
  
  <p>The data is only a few megabytes in size so I don't see how it can take up so much memory to process. </p>