title: Using sparse matrices/online learning in Naive Bayes (Python, scikit)
body: <p>I'm trying to do Naive Bayes on a dataset that has over 6,000,000 entries and each entry 150k features. I've tried to implement the code from the following link:
  <a href="https://stackoverflow.com/questions/10098533/implementing-bag-of-words-naive-bayes-classifier-in-nltk">Implementing Bag-of-Words Naive-Bayes classifier in NLTK</a></p>
  
  <p>The problem is (as I understand), that when I try to run the train-method with a dok_matrix as it's parameter, it cannot find iterkeys (I've paired the rows with OrderedDict as labels):</p>
  
  <pre><code>Traceback (most recent call last):
    File "skitest.py", line 96, in &lt;module&gt;
      classif.train(add_label(matr, labels))
    File "/usr/lib/pymodules/python2.6/nltk/classify/scikitlearn.py", line 92, in train
      for f in fs.iterkeys():
    File "/usr/lib/python2.6/dist-packages/scipy/sparse/csr.py", line 88, in __getattr__
      return _cs_matrix.__getattr__(self, attr)
    File "/usr/lib/python2.6/dist-packages/scipy/sparse/base.py", line 429, in __getattr__
      raise AttributeError, attr + " not found"
  AttributeError: iterkeys not found
  </code></pre>
  
  <p>My question is, is there a way to either avoid using a sparse matrix by teaching the classifier entry by entry (online), or is there a sparse matrix format I could use in this case efficiently instead of dok_matrix? Or am I missing something obvious?</p>
  
  <p>Thanks for anyone's time. :)</p>
  
  <p>EDIT, 6th sep:</p>
  
  <p>Found the iterkeys, so atleast the code runs. It's still too slow, as it has taken several hours with a dataset of the size of 32k, and still hasn't finished. Here's what I got at the moment:</p>
  
  <pre><code>matr = dok_matrix((6000000, 150000), dtype=float32)
  labels = OrderedDict()
  
  #collect the data into the matrix
  
  pipeline = Pipeline([('nb', MultinomialNB())])
  classif = SklearnClassifier(pipeline)
  
  add_label = lambda lst, lab: [(lst.getrow(x).todok(), lab[x])
                                for x in xrange(lentweets-foldsize)] 
  
  classif.train(add_label(matr[:(lentweets-foldsize),0], labels))
  readrow = [matr.getrow(x + foldsize).todok() for x in xrange(lentweets-foldsize)]
  data = np.array(classif.batch_classify(readrow))
  </code></pre>
  
  <p>The problem might be that each row that is taken doesn't utilize the sparseness of the vector, but goes through each of the 150k entry. As a continuation for the issue, does anyone know how to utilize this Naive Bayes with sparse matrices, or is there any other way to optimize the above code?</p>