title: Spark (scala) dataframes - Check whether strings in column contain any items from a set
body: <p>I'm pretty new to scala and spark and I've been trying to find a solution for this issue all day - it's doing my head in. I've tried 20 different variations of the following code and keep getting <code>type mismatch</code> errors when I try to perform calculations on a column.</p>
  
  <p>I have a spark dataframe, and I wish to check whether each string in a particular column contains any number of words from a pre-defined <code>List</code> (or <code>Set</code>) of words.</p>
  
  <p>Here is some example data for replication:</p>
  
  <pre><code>// sample data frame
  val df = Seq(
        (1, "foo"),
        (2, "barrio"),
        (3, "gitten"),
        (4, "baa")).toDF("id", "words")
  
  // dictionary Set of words to check 
  val dict = Set("foo","bar","baaad")
  </code></pre>
  
  <p>Now, i am trying to create a third column with the results of a comparison to see if the strings in the <code>$"words"</code> column within them contain any of the words in the <code>dict</code> Set of words. So the result should be:</p>
  
  <pre><code>+---+-----------+-------------+
  | id|      words|   word_check| 
  +---+-----------+-------------+
  |  1|        foo|         true|     
  |  2|      bario|         true|
  |  3|     gitten|        false|
  |  4|        baa|        false|
  +---+-----------+-------------+
  </code></pre>
  
  <p>First, I tried to see if i could do it natively without using UDFs, since the dict Set will actually be a large dictionary of > 40K words, and as I understand it this would be more efficient than a UDF:</p>
  
  <pre><code>df.withColumn("word_check", dict.exists(d =&gt; $"words".contains(d)))
  </code></pre>
  
  <p>But i get the error:</p>
  
  <pre><code>type mismatch;
  found   : org.apache.spark.sql.Column
  required: Boolean
  </code></pre>
  
  <p>I have also tried to create a UDF to do this (using both <code>mutable.Set</code> and <code>mutable.WrappedArray</code> to describe the Set - not sure which is correct but neither work):</p>
  
  <pre><code>val checker: ((String, scala.collection.mutable.Set[String]) =&gt; Boolean) = (col: String, array: scala.collection.mutable.Set[String] ) =&gt;  array.exists(d =&gt; col.contains(d))
  
  val udf1 = udf(checker)
  
  df.withColumn("word_check", udf1($"words", dict )).show()
  </code></pre>
  
  <p>But get another type mismatch:</p>
  
  <pre><code> found   : scala.collection.immutable.Set[String]
   required: org.apache.spark.sql.Column
  </code></pre>
  
  <p>If the set was a fixed number, I should be able to use <code>Lit(Int)</code> in the expression? But I don't really understand performing more complex functions on a column by mixing different data types works in scala. </p>
  
  <p>Any help greatly appreciated, especially if it can be done efficiently (it is a large df of > 5m rows).</p>