title: Efficient jaccard similarity DocumentTermMatrix
body: <p>I want a way to efficiently calculate Jaccard similarity between documents of a <code>tm::DocumentTermMatrix</code>.  I can do something similar for cosine similarity via the <strong>slam</strong> package as shown in <a href="https://stackoverflow.com/a/29755756/1000343">this answer.</a> I came across <a href="https://stats.stackexchange.com/a/89947/7482">another question and response</a> on CrossValidated that was R specific but about matrix algebra not necessarily the most efficient route.  I tried to implement that solution with more efficient <strong>slam</strong> functions but do not get the same solution as when I use a less efficient approach of coercing the DTM to a matrix and using <code>proxy::dist</code>.</p>
  
  <p>How can I efficiently calculate Jaccard similarity between documents of a large DocumentTermMatrix in R?</p>
  
  <p><strong>#Data &amp; Pacages</strong></p>
  
  <pre><code>library(Matrix);library(proxy);library(tm);library(slam);library(Matrix)
  
  mat &lt;- structure(list(i = c(1L, 2L, 1L, 2L, 1L, 2L, 1L, 2L, 3L, 1L, 
      2L, 3L, 3L, 3L, 4L, 4L, 4L, 4L), j = c(1L, 1L, 2L, 2L, 3L, 3L, 
      4L, 4L, 4L, 5L, 5L, 6L, 7L, 8L, 9L, 10L, 11L, 12L), v = c(1, 
      1, 1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1), nrow = 4L, 
          ncol = 12L, dimnames = structure(list(Docs = c("1", "2", 
          "3", "4"), Terms = c("computer", "is", "fun", "not", "too", 
          "no", "it's", "dumb", "what", "should", "we", "do")), .Names = c("Docs", 
          "Terms"))), .Names = c("i", "j", "v", "nrow", "ncol", "dimnames"
      ), class = c("DocumentTermMatrix", "simple_triplet_matrix"), weighting = c("term frequency", 
      "tf"))
  </code></pre>
  
  <p><strong>#Inefficient Calculation (expected output)</strong></p>
  
  <pre><code>proxy::dist(as.matrix(mat), method = 'jaccard')
  
  ##       1     2     3
  ## 2 0.000            
  ## 3 0.875 0.875      
  ## 4 1.000 1.000 1.000
  </code></pre>
  
  <p><strong>#My Attempt</strong></p>
  
  <pre><code>A &lt;- slam::tcrossprod_simple_triplet_matrix(mat)
  im &lt;- which(A &gt; 0, arr.ind=TRUE)
  b &lt;- slam::row_sums(mat)
  Aim &lt;- A[im]
  
  stats::as.dist(Matrix::sparseMatrix(
        i = im[,1],
        j = im[,2],
        x = Aim / (b[im[,1]] + b[im[,2]] - Aim),
        dims = dim(A)
  ))
  
  ##     1   2   3
  ## 2 2.0        
  ## 3 0.1 0.1    
  ## 4 0.0 0.0 0.0
  </code></pre>
  
  <p>Outputs do not match.</p>
  
  <p>FYI Here is the original text:</p>
  
  <pre><code>c("Computer is fun. Not too fun.", "Computer is fun. Not too fun.", 
      "No it's not, it's dumb.", "What should we do?")
  </code></pre>
  
  <p>I'd expect elements 1 &amp; 2 to be 0 distance and element 3 to be closer to  element 1 than element 1 and 4 (I'd expect furthest distance as no words are shared) as seen in the <code>proxy::dist</code> solution.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>Note that even on a medium sized DTM the matrix becomes huge.  Here's an example with the <strong>vegan</strong> package.  Note 4 minutes to solve where as the cosine similarity is ~5 seconds.</p>
  
  <pre><code>library(qdap); library(quanteda);library(vegan);library(slam)
  x &lt;- quanteda::convert(quanteda::dfm(rep(pres_debates2012$dialogue), stem = FALSE, 
          verbose = FALSE, removeNumbers = FALSE), to = 'tm')
  
  
  ## &lt;&lt;DocumentTermMatrix (documents: 2912, terms: 3368)&gt;&gt;
  ## Non-/sparse entries: 37836/9769780
  ## Sparsity           : 100%
  ## Maximal term length: 16
  ## Weighting          : term frequency (tf)
  
  tic &lt;- Sys.time()
  jaccard_dist_mat &lt;- vegan::vegdist(as.matrix(x), method = 'jaccard')
  Sys.time() - tic #Time difference of 4.01837 mins
  
  tic &lt;- Sys.time()
  tdm &lt;- t(x)
  cosine_dist_mat &lt;- 1 - crossprod_simple_triplet_matrix(tdm)/(sqrt(col_sums(tdm^2) %*% t(col_sums(tdm^2))))
  Sys.time() - tic #Time difference of 5.024992 secs
  </code></pre>