title: How to compare two columns in two different dataframes in pyspark
body: <p>i want to compare "pitid" in one dataframe1 with "pitid" of another dataframe2 and want to extract the rows that are missing in dataframe1.</p>
  
  <p>dataframe1:</p>
  
  <pre><code> | id|marks|name|      pitid|
  +---+-----+----+-----------+
  |  1|    1|  FR| 1496875194|
  |  2|    1|  US| -744211593|
  |  5|    2|  DE|-1433680238|
  |  4|    1|  DE| -366408878|
  |  3|    3|  DE|  526286357|
  +---+-----+----+-----------+
  </code></pre>
  
  <p>dataframe2:</p>
  
  <pre><code>| id|marks|name|      pitid|
  +---+-----+----+-----------+
  |  1|    1|  FR| 1496875194|
  |  7|    9|  HY| -816101137|
  |  6|    5|  FE| 1044793796|
  |  2|    1|  US| -744211593|
  |  5|    2|  DE|-1433680238|
  |  4|    1|  DE| -366408878|
  |  3|    3|  DE|  526286357|
  +---+-----+----+-----------+
  </code></pre>
  
  <p>expected output:</p>
  
  <pre><code>|  7|    9|  HY| -816101137|
  |  6|    5|  FE| 1044793796|
  </code></pre>