title: Removing duplicate columns after a DF join in Spark
body: <p>When you join two DFs with similar column names:</p>
  <pre><code>df = df1.join(df2, df1['id'] == df2['id'])
  </code></pre>
  <p>Join works fine but you can't call the <code>id</code> column because it is ambiguous and you would get the following exception:</p>
  <blockquote>
  <p>pyspark.sql.utils.AnalysisException: &quot;Reference 'id' is ambiguous,
  could be: id#5691, id#5918.;&quot;</p>
  </blockquote>
  <p>This makes <code>id</code> not usable anymore...</p>
  <p>The following function solves the problem:</p>
  <pre><code>def join(df1, df2, cond, how='left'):
      df = df1.join(df2, cond, how=how)
      repeated_columns = [c for c in df1.columns if c in df2.columns]
      for col in repeated_columns:
          df = df.drop(df2[col])
      return df
  </code></pre>
  <p>What I don't like about it is that I have to iterate over the column names and delete them why by one. This looks really clunky...</p>
  <p>Do you know of any other solution that will either join and remove duplicates more elegantly or delete multiple columns without iterating over each of them?</p>