title: Drop multiple duplicated columns after left join w/ dataframes?
body: <p>I'm working on a script using Pyspark and Spark SQL. However, after running a <code>LEFT-JOIN</code> on my <code>base_df</code> and <code>inc_df</code> dateframes,  all of my columns were duplicated.</p>
  <p>I've figured out why the columns were duplicated, but I'm now receiving <code>type errors</code> while trying to <code>DROP</code> those duplicated columns.</p>
  <p>I've tried this solution as well.... <code>a.join(b, 'id')</code></p>
  <p>Here's part of my code below..</p>
  <pre><code>
  ''' uc_df = (base_df
  
           .join(inc_df,
  
                 (inc_df.src_sys_id == base_df.src_sys_id) &amp; (inc_df.fleet_acct_nbr == base_df.fleet_acct_nbr),
  
                 &quot;left_outer&quot;)
  
           .filter(inc_df.src_sys_id.isNull())
  
           .drop(base_df.region_cd,
  
                 base_df.fleet_acct_exctv_cd,
  
                 base_df.fleet_acct_nbr,
  
                 base_df.fleet_acct_exctv_nm,
  
                 base_df.fleet_acct_exctv_first_nm,
  
                 base_df.fleet_acct_exctv_last_nm,
  
                 base_df.fleet_acct_exctv_mid_nm,
  
                 base_df.fleet_acct_nm,
  
                 base_df.meas_cnt,
  
                 base_df.dw_anom_flg,
  
                 base_df.dw_mod_ts,
  
                 base_df.src_sys_iud_cd,
  
                 base_df.dw_job_id)
  
         
  
          )
  
  </code></pre>
  <p>I received the following error:</p>
  <pre><code>Type Error: Each column in the param list should be a string
  
  </code></pre>