title: Reducing with a bloom filter
body: <p>I would like to get a fast approximate set membership, based on a String-valued function applied to a large Spark RDD of String Vectors (~1B records). Basically the idea would be to reduce into a <a href="https://en.wikipedia.org/wiki/Bloom_filter" rel="nofollow">Bloom filter</a>. This bloom filter could then be broadcasted to the workers for further use.</p>
  
  <p>More specifically, I currently have</p>
  
   
  
  <pre><code>rdd: RDD[Vector[String]]
  f: Vector[String] =&gt; String
  val uniqueVals = rdd.map(f).distinct().collect()
  val uv = sc.broadcast(uniqueVals)
  </code></pre>
  
  <p>But <code>uniqueVals</code> is too large to be practical, and I would like to replace it with something of smaller (and known) size, i.e. a bloom filter. </p>
  
  <p>My questions: </p>
  
  <ul>
  <li><p>is it possible to reduce into a Bloom filter, or do I have to collect first, and then construct it in the driver?</p></li>
  <li><p>is there a mature Scala/Java Bloom filter implementation available that would be suitable for this?</p></li>
  </ul>