title: Python pandas: Generate Document-Term matrix from whitespace delimited '.dat' file
body: <p>I'm using Python to attempt to rank documents using an <a href="https://en.wikipedia.org/wiki/Okapi_BM25" rel="nofollow">Okapi BM25 model</a>.</p>
  
  <p>I think that I can calculate some of the terms required for the <code>Score(D,Q)</code> such as the IDF (Inverse Document Frequency) in a more efficient way (i.e: Counting all non-zero rows for a particular term (column)). Furthermore, I can add a new column to the matrix for the actual Score and then sort by this to rank documents.</p>
  
  <p>The document term vectors are stored in a <code>.dat</code> file which is structured like the following:</p>
  
  <pre><code>D1 7:10 2:5
  D2 1:2 3:4
  </code></pre>
  
  <p>where <code>D1</code> is a document ID and <code>7:10</code> represents the term with ID <code>7</code> appearing <code>10</code> times </p>
  
  <p>At the moment, I am reading it into a list of lists using the following code:</p>
  
  <pre><code>fname = "dtv.dat"
  f = open(fname, "r")
  l = [x.strip(" \n").split(" ") for x in f.readlines()]
  </code></pre>
  
  <p>which yields the following output for the given example:</p>
  
  <pre><code>[['D1', '7:10', '2:5'],['D2' '1:2', '3:4']]
  </code></pre>
  
  <p>Given this list of list format, what is the most efficient way to convert this to a Python pandas DataFrame similar to the following:</p>
  
  <pre><code>0      1     2      3      7
  D1     0     5      0      10    
  D2     2     0      4      0
  </code></pre>