title: spark sql dynamic filter condition
body: <p>How can I construct a boolean filter condition dynamically in spark sql?
  Having:</p>
  
  
  
  <pre class="lang-scala prettyprint-override"><code>val d = Seq(1, 2, 3, 5, 6).toDF
  d.filter(col("value") === 1 or col("value") === 3).show
  </code></pre>
  
  <p>How can I replicate this dynamically:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val desiredThings = Seq(1,3)
  </code></pre>
  
  <p>I try to build the filter:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val myCondition = desiredThings.map(col("value") === _)
  d.filter(myCondition).show
  </code></pre>
  
  <p>but fail with:</p>
  
  <pre class="lang-scala prettyprint-override"><code>overloaded method value filter with alternatives:
  org.apache.spark.api.java.function.FilterFunction[org.apache.spark.sql.Row]
   cannot be applied to (Seq[org.apache.spark.sql.Column])
  </code></pre>
  
  <p>When executing</p>
  
  <pre class="lang-scala prettyprint-override"><code>d.filter(myCondition).show
  </code></pre>
  
  <p>Also when experimenting with fold left:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val myCondition = desiredThings.foldLeft()((result, entry) =&gt; result &amp;&amp; col(c.columnCounterId) === entry)
  </code></pre>
  
  <p>I have compile errors.</p>
  
  <p>How can I adapt the code to dynamically generate the filter predicate?</p>