title: Combinations And Mass Set Intersections With Apache Spark (Pyspark)
body: <p><strong>The Challenge:</strong>
  The data is an RDD in the shape of [(u, p), (u, p), ...] where u and p are both strings. My challenging desired output is [((p,p), u), ..] where the key in each element is a combination of all p's that share a u.</p>
  
  <p><strong>Already Tried:</strong></p>
  
  <ol>
  <li>Using cartesian and filter to simulate combination.</li>
  <li>Tried to use itertools.combinations in Python for each element and then flatMap, like something like this:<code>[(tuple(sorted(e)), x[0]) for e in combinations(x[1].split(','),2)])</code></li>
  </ol>
  
  <p>The problem with what I already tried is that the executor nodes seem to go down, presumably from memory consumption. </p>
  
  <p>Any suggestions?</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>Here's some more info and background for the question:</p>
  
  <p>I'm looking to find the overlap count/cardinality of the intersection for every two "p"s where a "p" is a collection id and "u" is a member of a collection. </p>
  
  <p>The input is a huge list of records of relationships between a "p" and a "u", which can be for example: 
  <code>[(u1,p1), (u2, p2), (u1, p2), (u2, p1), (u1, p3)]</code></p>
  
  <p>The desired output is: <code>[((p1,p2), 2), ((p1, p3), 1), ((p2, p3), 1)]</code> 
  (note combinations and not permutations)</p>
  
  <p>A few more details about the input data:</p>
  
  <ol>
  <li>There are 50k - 100k different "p"s.</li>
  <li>A "p" can have a few hundred million unique "u"s.</li>
  <li>There can be duplicate rows of input data. In other words multiple instances of <code>(u,p)</code> but that doesn't change the problem when it comes to treating the list of "u"s as a set for a given "p" for the intersection step.</li>
  </ol>
  
  <p>As for hardware, I'm using about 41 m3.xlarges on EMR (Spark 1.3.1 on Yarn) and kicking the shell off this way:
  <code>./pyspark --master yarn-client --driver-memory 4G --executor-memory 3G --num-executors 160</code></p>