title: Spark: How do I pass a PartialFunction to a DStream?
body: <p>I'm trying to pass a partial function to the union of all the RDDs captured in a DStream batch over a sliding window. Lets say I construct a window operation over 10 seconds on a stream discretized into 1 second batches:</p>
  
  <pre><code>val ssc = new StreamingContext(new SparkConf(), Seconds(1))
  val stream = ssc.socketStream(...)
  val window = stream.window(Seconds(10))
  </code></pre>
  
  <p>My <code>window</code> will have K many RDDs. I want to use <code>collect(f: PartialFunction[T, U])</code> on the union of all K of these RDDs. I could call the union operator <code>++</code> using <code>foreachRDD</code>, but I want to return an <code>RDD</code> not a <code>Unit</code> and avoid side effects. </p>
  
  <p>What I'm looking for is a reducer like </p>
  
  <pre><code>def reduce(f: (RDD[T], RDD[T]) ⇒ RDD[T]): RDD[T]
  </code></pre>
  
  <p>on a <code>DStream</code> that I can use like so:</p>
  
  <pre><code>window.reduce(_ ++ _).transform(_.collect(myPartialFunc))
  </code></pre>
  
  <p>But this is not available in the Spark Streaming API. </p>
  
  <p>Does anyone have any good ideas for combining RDDs captured in a stream into a single RDD so I can pass in a partial function? Or for implementing my own RDD reducer? Perhaps this feature is coming in a subsequent Spark release?</p>