title: Creating a sparse matrix from a TermDocumentMatrix
body: <p>I've created a <code>TermDocumentMatrix</code> from the <code>tm</code> library in R. It looks something like this:</p>
  
  <pre><code>&gt; inspect(freq.terms)
  
  A document-term matrix (19 documents, 214 terms)
  
  Non-/sparse entries: 256/3810
  Sparsity           : 94%
  Maximal term length: 19 
  Weighting          : term frequency (tf)
  
  Terms
  Docs abundant acid active adhesion aeropyrum alternative
    1         0    0      1        0         0           0
    2         0    0      0        0         0           0
    3         0    0      0        1         0           0
    4         0    0      0        0         0           0
    5         0    0      0        0         0           0
    6         0    1      0        0         0           0
    7         0    0      0        0         0           0
    8         0    0      0        0         0           0
    9         0    0      0        0         0           0
    10        0    0      0        0         1           0
    11        0    0      1        0         0           0
    12        0    0      0        0         0           0
    13        0    0      0        0         0           0
    14        0    0      0        0         0           0
    15        1    0      0        0         0           0
    16        0    0      0        0         0           0
    17        0    0      0        0         0           0
    18        0    0      0        0         0           0
    19        0    0      0        0         0           1
  </code></pre>
  
  <p>This is just a small sample of the matrix; there are actually 214 terms that I'm working with. On a small scale, this is fine. If I want to convert my <code>TermDocumentMatrix</code> into an ordinary matrix, I'd do:</p>
  
  <pre><code>data.matrix &lt;- as.matrix(freq.terms)
  </code></pre>
  
  <p>However the data that I've displayed above is just a subset of my overall data. My overall data has probably at least 10,000 terms. When I try to create a TDM from the overall data, I run an error:</p>
  
  <pre><code>&gt; Error cannot allocate vector of size n Kb
  </code></pre>
  
  <p>So from here, I'm looking into alternative ways of finding efficient memory allocation for my tdm. </p>
  
  <p>I tried turning my tdm into a sparse matrix from the <code>Matrix</code> library but ran into the same problem. </p>
  
  <p>What are my alternatives at this point? I feel like I should be investigating one of:</p>
  
  <ul>
  <li><code>bigmemory</code>/<code>ff</code> packages as talked about <a href="http://www.bytemining.com/2010/08/taking-r-to-the-limit-part-ii-large-datasets-in-r/" rel="nofollow noreferrer">here</a>  (although the <code>bigmemory</code> package doesn't seem available for Windows at the moment)</li>
  <li>the <code>irlba</code> package for computing partials SVD of my tdm as mentioned <a href="https://stackoverflow.com/questions/4951286/svd-for-sparse-matrix-in-r">here</a></li>
  </ul>
  
  <p>I've experimented with functions from both libraries but can't seem to arrive at anything substantial. Does anyone know what the best way forward is? I've spent so long fiddling around with this that I thought I'd ask people who have much more experience than myself working with large datasets before I waste even more time going in the wrong direction.</p>
  
  <p>EDIT: changed 10,00 to 10,000. thanks @nograpes.</p>