title: Filtering some data from Spark dataframe based on CSV records
body: <p>I have a <code>CSV</code> file containing some words. Total no. of words in csv file won't exceed 50k records.</p>
  
  <p>I have a <code>Spark Dataframe</code> created from a <code>JSON</code> file having <code>keywords</code> column. What I need to do is filter out the records from dataframe whose <code>keywords</code> column value matches with the value present in CSV file. Here, Matches means if the word in csv file appears in dataframe column.</p>
  
  <p>Taking an example, let's say there is a word <code>"baby toys"</code> in csv file and <code>spark dataframe</code> looks like this</p>
  
  <pre><code>***Keywords***
  new baby toys
  baby toys for all
  costly baby toys price
  baby has toys
  </code></pre>
  
  <p>In the above first 3 rows should be filtered out as all having the word <code>baby toys</code> in sequence.</p>
  
  <p>To implement, I am doing something like this.</p>
  
  <pre><code>1. Reading csv file and creating a dataframe.
  
  2. Collecting all the words as an array of strings from dataframe created above as
  val negativeKeywords = csvDF.distinct.map(x =&gt; x.getString(0)).collect()
  
  3. Creating a UDF to match the words - one from negative list created above and other from Spark dataframe(created in step 4)
   val udfmatch= udf((x: String) =&gt; {
        val loop = new Breaks
        var check = false
        loop.breakable{
          for(s &lt;- negativeKeywords){
            if(x.contains(s)){
              check = true
              loop.break
            }
          }
        }
        check
    })
  
  4. Created spark dataframe from JSON file. 
  5. Filter from the above JSON dataframe using UDF defined above. 
     sparkDf.filter(udfmatch(col("keyword_text")))
  </code></pre>
  
  <p>In the above, I am iterating over the entire <code>csv</code> words list(until I find it) for every keyword present in <code>Spark dataframe</code>, which i think is not right and time consuming.
  Can someone please suggest a better way.</p>