title: Spark: what's the best strategy for joining a 2-tuple-key RDD with single-key RDD?
body: <p>I have two RDD's that I want to join and they look like this:</p>
  
  <pre><code>val rdd1:RDD[(T,U)]
  val rdd2:RDD[((T,W), V)]
  </code></pre>
  
  <p>It happens to be the case that the key values of <code>rdd1</code> are unique and also that the tuple-key values of <code>rdd2</code> are unique.  I'd like to join the two data sets so that I get the following rdd:</p>
  
  <pre><code>val rdd_joined:RDD[((T,W), (U,V))]
  </code></pre>
  
  <p>What's the most efficient way to achieve this?  Here are a few ideas I've thought of.  </p>
  
  <p>Option 1:</p>
  
  <pre><code>val m = rdd1.collectAsMap
  val rdd_joined = rdd2.map({case ((t,w), u) =&gt; ((t,w), u, m.get(t))})
  </code></pre>
  
  <p>Option 2:</p>
  
  <pre><code>val distinct_w = rdd2.map({case ((t,w), u) =&gt; w}).distinct
  val rdd_joined = rdd1.cartesian(distinct_w).join(rdd2)
  </code></pre>
  
  <p>Option 1 will collect all of the data to master, right?  So that doesn't seem like a good option if rdd1 is large (it's relatively large in my case, although an order of magnitude smaller than rdd2).  Option 2 does an ugly distinct and cartesian product, which also seems very inefficient.  Another possibility that crossed my mind (but haven't tried yet) is to do option 1 and broadcast the map, although it would be better to broadcast in a "smart" way so that the keys of the map are co-located with the keys of <code>rdd2</code>.</p>
  
  <p>Has anyone come across this sort of situation before?  I'd be happy to have your thoughts.</p>
  
  <p>Thanks!</p>