title: Scala Spark filter inside map
body: <p>I want to efficiently filter an RDD while mapping it. Is that possible?</p>
  
  <p>Here is pseudocode for what I want to do:</p>
  
  <pre><code>for element in rdd:
      val opt = f(element)
      if (opt.nonEmpty) add_pair(opt.get, element)
  </code></pre>
  
  <p>Here is a hacky way to implement the pseudocode in Scala Spark:</p>
  
  <pre><code>rdd.map(element =&gt; (
      f(element).getOrElse(99),
      element
  )).filter(tuple =&gt; tuple._1 != 99)
  </code></pre>
  
  <p>I wasn't able to find clean syntax to do this, so I first mapped all elements, and then filtered out the ones I don't want. Note that potentially expensive call <code>f(element)</code> is only computed once. If I were to filter elements before mapping (which would look more clean), then I would end up calling <code>f</code> twice, which is inefficient.</p>
  
  <p>Please do not flag this as a duplicate. While there are similar questions, none of them actually answer this question. For example, <a href="https://stackoverflow.com/questions/28843591/spark-filter-within-map">this</a> potential duplicate would call <code>f</code> twice, which is inefficient, and therefore does not answer this question.</p>