title: Compare String Values in 2 Spark Dataframes
body: <p>I have 2 dataframes named - brand_name and poi_name.</p>
  
  <p><strong>Dataframe 1(brand_name):-</strong></p>
  
  <pre><code>+-------------+
  |brand_stop[0]|
  +-------------+
  |TOASTMASTERS |
  |USBORNE      |
  |ARBONNE      |
  |USBORNE      |
  |ARBONNE      |
  |ACADEMY      |
  |ARBONNE      |
  |USBORNE      |
  |USBORNE      |
  |PILLAR       |
  +-------------+
  </code></pre>
  
  <p><strong>Dataframe 2:-(poi_name)</strong></p>
  
  <pre><code>+---------------------------------------+
  |Name                                   |
  +---------------------------------------+
  |TOASTMASTERS DISTRICT 48               |
  |USBORNE BOOKS AND MORE                 |
  |ARBONNE                                |
  |USBORNE BOOKS AT HOME                  |
  |ARBONNE                                |
  |ACADEMY, LTD.                          |
  |ARBONNE                                |
  |USBORNE BOOKS AT HOME                  |
  |USBORNE BOOKS &amp; MORE                   |
  |PILLAR TO POST HOME INSPECTION SERVICES|
  +---------------------------------------+
  </code></pre>
  
  <p>I want to check whether the strings in brand_stop column of dataframe 1 are present in Name column of dataframe 2. The matching should be done row wise and then if there is a successful match, that particular record should be stored in a new column. </p>
  
  <p>I have tried filtering the dataframe using Join:-</p>
  
  <pre><code>from pyspark.sql.functions import udf, col 
  from pyspark.sql.types import BooleanType
  
  contains = udf(lambda s, q: q in s, BooleanType())
  
  like_with_python_udf = (poi_names.join(brand_names1)
      .where(contains(col("Name"), col("brand_stop[0]")))
      .select(col("Name")))
  like_with_python_udf.show()
  </code></pre>
  
  <p>But this shows an error</p>
  
  <pre><code>"AnalysisException: u'Detected cartesian product for INNER join between logical plans"
  </code></pre>
  
  <p>I am new to PySpark. Please help me with this.</p>
  
  <p>Thank you</p>