title: Spark broadcast vs join
body: <p>I have a large RDD (rdd_1) and a filtered subset (rdd_2) of it. I want to join rdd_1 and rdd_2 on a different field.</p>
  
  <p>Let's say records are of format {'first_name':&lt;>, 'last_name':&lt;>}. We want to find all names that have the same last name as all the 'jack's.</p>
  
  <pre><code>names = sc.textfile(RAW_DATA)
  jack = names.filter(lambda v: v['first_name'] == 'jack')
  </code></pre>
  
  <p><strong>Option 1</strong></p>
  
  <pre><code>jack_last_names = jack.map(operator.itergetter('last_name').distinct().collect()
  last_names_bc = sc.broadcast(set(jack_last_names))
  final = names.filter(lambda v:v['last_name'] in last_names_bc.value)
  </code></pre>
  
  <p>Currently, I broadcast rdd_2 and filter rdd_1 by it. Trouble is that in order to broadcast rdd_2, I have to first collect() it on the driver and it causes driver to run out of memory.</p>
  
  <p>Is there a way to broadcast an RDD without first collect()ing it on the driver?</p>
  
  <p><strong>Option 2</strong></p>
  
  <pre><code>final = jack.keyBy(operator.itemgetter('last_name').join(names.keyBy(operator.itemgetter('last_name')
  </code></pre>
  
  <p>My other option is rdd_1.join(rdd_2) but rdd_1 is way too big to shuffle.</p>
  
  <p>When we run rdd_1.join(rdd_2) do both rdd_1 and rdd_2 get hash partitioned and shuffled?</p>
  
  <p>Thanks!</p>