title: scikit-learn DBSCAN memory usage
body: <p><strong>UPDATED:</strong> In the end, the solution I opted to use for clustering my large dataset was one suggested by Anony-Mousse below. That is, using ELKI's DBSCAN implimentation to do my clustering rather than scikit-learn's. It can be run from the command line and with proper indexing, performs this task within a few hours. Use the GUI and small sample datasets to work out the options you want to use and then go to town. Worth looking into. Anywho, read on for a description of my original problem and some interesting discussion.</p>
  
  <p>I have a dataset with ~2.5 million samples, each with 35 features (floating point values) that I'm trying to cluster. I've been trying to do this with scikit-learn's implementation of DBSCAN, using the Manhattan distance metric and a value of epsilon estimated from some small random samples drawn from the data. So far, so good. (here is the snippet, for reference)</p>
  
  <pre><code>db = DBSCAN(eps=40, min_samples=10, metric='cityblock').fit(mydata)
  </code></pre>
  
  <p>My issue at the moment is that I easily run out of memory. (I'm currently working on a machine with 16 GB of RAM)</p>
  
  <p>My question is, is DBSCAN calculating the pairwise distance matrix on the fly as it runs, and that's what's gobbling up my memory? (2.5 million ^ 2) * 8 bytes is obviously stupidly large, I would understand that. Should I not be using the <code>fit()</code> method? And more generally, is there a way around this issue, or am I generally barking up the wrong tree here?</p>
  
  <p>Apologies if the answer winds up being obvious. I've been puzzling over this for a few days. Thanks!</p>
  
  <p>Addendum: Also if anyone could explain the difference between <code>fit(X)</code> and <code>fit_predict(X)</code> to me more explicitly I'd also appreciate that--I'm afraid I just don't quite get it.</p>
  
  <p>Addendum #2: To be sure, I just tried this on a machine with ~550 GB of RAM and it still blew up, so I feel like DBSCAN is likely trying to make a pairwise distance matrix or something I clearly don't want it to do. I guess now the big question is how to stop that behavior, or find other methods that might suit my needs more. Thanks for bearing with me here. </p>
  
  <p>Addendum #3(!): I forgot to attach the traceback, here it is,</p>
  
  <pre><code>Traceback (most recent call last):
    File "tDBSCAN.py", line 34, in &lt;module&gt;
      db = DBSCAN(eps=float(sys.argv[2]), min_samples=10, metric='cityblock').fit(mydata)
    File "/home/jtownsend/.local/lib/python2.6/site-packages/sklearn/base.py", line 329, in fit_predict
      self.fit(X)
    File "/home/jtownsend/.local/lib/python2.6/site-packages/sklearn/cluster/dbscan_.py", line 186, in fit
      **self.get_params())
    File "/home/jtownsend/.local/lib/python2.6/site-packages/sklearn/cluster/dbscan_.py", line 69, in dbscan
      D = pairwise_distances(X, metric=metric)
    File "/home/jtownsend/.local/lib/python2.6/site-packages/sklearn/metrics/pairwise.py", line 651, in pairwise_distances
      return func(X, Y, **kwds)
    File "/home/jtownsend/.local/lib/python2.6/site-packages/sklearn/metrics/pairwise.py", line 237, in manhattan_distances
      D = np.abs(X[:, np.newaxis, :] - Y[np.newaxis, :, :])
  MemoryError
  </code></pre>