title: How to remove 'duplicate' rows from joining the same pyspark dataframe?
body: <p>I need to show a dataframe made by three columns. Two of them show the names of someone who worked in a common movie (indicated by code on the third code)</p>
  
  <p>here's my code for the query:</p>
  
  <pre><code>name_data_df3 = movie_join_oscar_join_name.filter('Nazione in ("USA", "England", "France", "Canada", "Italy", "Austria")').select('knownForTitle','person').distinct()
  art_pre_edge_data1 = name_data_df3.select(col('person').alias('artist1'), 'knownForTitle', )
  art_pre_edge_data2 = art_pre_edge_data1.select(col('artista').alias('artist2'), col('knownForTitle').alias('knownForTitle2'))
  art_edge_data = art_pre_edge_data1.join(art_pre_edge_data2, (art_pre_edge_data1.knownForTitle==art_pre_edge_data2.knownForTitle2) &amp; (art_pre_edge_data1.artista!=art_pre_edge_data2.artista2))
  art_edge_data = art_edge_data.drop(art_pre_edge_data2.knownForTitle2)
  art_edge_data.select('*').show()
  </code></pre>
  
  <p>and here is the result:</p>
  
  <pre><code>+-----------------+-------------+-----------------+
  |          artist1|knownForTitle|         artist2|
  +-----------------+-------------+-----------------+
  |   Robert Redford|    tt0070735|  George Roy Hill|
  |  George Roy Hill|    tt0070735|   Robert Redford|
  | Lionel Barrymore|    tt0022958|    Wallace Beery|
  | Lionel Barrymore|    tt0022958|    Joan Crawford|
  |    Wallace Beery|    tt0022958| Lionel Barrymore|
  |    Wallace Beery|    tt0022958|    Joan Crawford|
  |    Joan Crawford|    tt0022958| Lionel Barrymore|
  |    Joan Crawford|    tt0022958|    Wallace Beery|
  </code></pre>
  
  <p>how can I remove the duplicates? </p>