title: pointers in sparse matrix in python scipy
body: <p>I am trying to understand sparse matrix in scipy especially the <strong>csr_matrix format</strong> </p>
  
  <p>Suppose I have following texts </p>
  
  <pre><code> docs = ['hello  world hello', 'goodbye cruel world']
  </code></pre>
  
  <p>I tokenize them and get a list of dictionaries with token occurences and a  dictionary with token_ids. </p>
  
  <pre><code>ids_token = {0: 'world', 1: 'hello', 2: 'cruel', 3: 'goodbye'}
  token_counts = [{0: 1, 1: 2}, {0: 1, 2: 1, 3: 1}]
  </code></pre>
  
  <p><strong>How can I transform the token_counts in  csr_matrix ?</strong></p>
  
  <p>Here is what I tried so far:</p>
  
  <pre><code>data = [item for sublist in token_counts for item in sublist.values()]
  print 'data:', data
  
  indices = [item for sublist in token_counts for item in sublist.keys()]
  print 'indices:', indices 
  
  indptr  = [0] + [len(item) for item in token_counts]
  print 'pointers:', indptr
  
  #now I create the matrix 
  sp_matrix = csr_matrix((data, indices, indptr), dtype=int)
  print sp_matrix.toarray()
  
  import pandas as pd 
  pd.DataFrame(sp_matrix.toarray().transpose(), index = ids_token.values())
  </code></pre>
  
  <p>the results is not what expect, which zeros in the last rows. </p>
  
  <p>I suspect that the problem is in the pointer indptr, what am I missing ? </p>
  
  <p>any help appreciated </p>
  
  <p><strong>updated</strong>
  this is what I would like to get </p>
  
  <pre><code>       doc0  doc11
  cruel   0   1
  goodbye 0   1
  hello   2   0
  world   1   1
  </code></pre>
  
  <p>P.S: the example is taken from the <a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html#scipy.sparse.csr_matrix" rel="nofollow">scipy documentation</a> </p>