title: How to compare Spark dataframe columns with another dataframe column values
body: <p>I have two dataframes as below:</p>
  
  <pre><code>df1: Which will have few values as below. This is dynamic.
  
  +--------------+
  |tags          |
  +--------------+
  |first_name    |
  |last_name     |
  |primary_email |
  |other_email   |
  +--------------+
  </code></pre>
  
  <p>df2: The second dataframe has few pre-defined combinations as below:</p>
  
  <pre><code>+---------------------------------------------------------------------------------------------+
  |combinations                                                                                 |
  +---------------------------------------------------------------------------------------------+
  |last_name, first_name, primary_email                                                         |
  |last_name, first_name, other_email                                                           |
  |last_name, primary_email, primary_phone                                                      |
  |last_name, primary_email, secondary_phone                                                    |
  |last_name, address_line1, address_line2,city_name, state_name,postal_code, country_code, guid|
  +---------------------------------------------------------------------------------------------+
  </code></pre>
  
  <p>Expected Result DF: 
  Now, I wanted to find that from my dataframe, is there any valid combinations I can make. The result should have all the valid combinations if it matches any with <code>combinations</code> dataframe.</p>
  
  <pre><code>resultDF:
  
  +---------------------------------------------------------------------------------------------+
  |combinations                                                                                 |
  +---------------------------------------------------------------------------------------------+
  |last_name, first_name, primary_email                                                         |
  |last_name, first_name, other_email                                                           |
  +---------------------------------------------------------------------------------------------+
  </code></pre>
  
  <p>I tried an approach of converting both the dataframes into list and try to compare it, but I always getting 0 combinations.</p>
  
  <p>The scala code I tried.</p>
  
  <pre><code>val combinationList = combinations.map(r =&gt; r.getString(0)).collect.toList
  
  var combList: Seq[Seq[String]]  = Seq.empty
  
      for (comb &lt;- combinationList) {
        var tmp: Seq[String] = Seq.empty
        tmp = tmp :+ comb
        combList = combList :+ tmp
      }
  
  val result = combList.filter(
        list =&gt; df1.filter(df1.col("tags").isin(list: _*)).count == list.size
      )
  
  println(result.size)
  </code></pre>
  
  <p>This is always return 0. The answer should be 2.</p>
  
  <p>Can someone guide me what is the best approach ?</p>