title: Memory efficient cartesian join in PySpark
body: <p>I have a large dataset of string ids, that can fit into memory on a single node in my spark cluster. The issue is that it consumes most of the memory for a single node.</p>
  
  <p>These ids are about 30 characters long. For example:</p>
  
  <pre><code>ids
  O2LWk4MAbcrOCWo3IVM0GInelSXfcG
  HbDckDXCye20kwu0gfeGpLGWnJ2yif
  o43xSMBUJLOKDxkYEQbAEWk4aPQHkm
  </code></pre>
  
  <p>I am looking to write to file a list of all of the pairs of ids. For example:</p>
  
  <pre><code>id1,id2
  O2LWk4MAbcrOCWo3IVM0GInelSXfcG,HbDckDXCye20kwu0gfeGpLGWnJ2yif
  O2LWk4MAbcrOCWo3IVM0GInelSXfcG,o43xSMBUJLOKDxkYEQbAEWk4aPQHkm
  HbDckDXCye20kwu0gfeGpLGWnJ2yif,O2LWk4MAbcrOCWo3IVM0GInelSXfcG
  # etc...
  </code></pre>
  
  <p>So I need to cross join the dataset on itself. I was hoping to do this on PySpark using a 10 node cluster, but it needs to be memory efficient.</p>