title: How to create a co-occurrence matrix from a Spark RDD
body: <p>I want to create a co-occurrence matrix from some tuples, see below:</p>
  
  <pre><code>val rdd = sc.parallelize(Array(
    Array("101","103","105"), Array("102","105"),Array("101","102","103","105"))
  val coocMatrix = new ArrayBuffer[(String, Int)]()
  
  // map
  rdd.collect().foreach(x =&gt; 
    { for(i &lt;- 0 to x.length-1)
    { for(j &lt;- i+1 to x.length-1)
    { coocMatrix += (x(i)+"#"+x(j), 1) }}
  
  // convert to rdd again
  val rdd2 = sc.parallelize(coocMatrix)
  
  // reduce
  vall matrix = rdd2.collect().groupByKey()
  </code></pre>
  
  <p>So we get following data finally</p>
  
  <pre><code>(101#103,2),(101#105,2),(102#105,2),(101#102,1),
  (103#105,2),(102#103,1),(102#105,1)
  </code></pre>
  
  <p>This algorithm is terribly slow because it's O(n*n) and infeasible when there 2 million tuples. Is there any algorithm to calculate this co-occurrence matrix?</p>