title: How to remove column duplication in PySpark DataFrame without declare column name where the columns are unknown to me?
body: <p>I have one source data:</p>
  <p>employee_id   employee_name  salary     designation<br />
  101           monali         70000      developer
  102           Amy            70000      developer
  103           neha           65000      tester</p>
  <p>I have a schema :
  [&quot;employee_id&quot;,&quot;employee_name&quot;,&quot;salary&quot;,&quot;designation&quot; ]</p>
  <p>Now I got 1 extra column in the source data which is unknown to me, so it is not in the schema/metadata.</p>
  <p>employee_id   employee_name  salary     designation    gender<br />
  101           monali         70000      developer
  102           Amy            70000      developer
  103           neha           65000      tester
  104           amar           76000      developer       male</p>
  <p>I need to write this extra column along with the records where the extra columns are present without selecting the valid columns. This column is ever changing, so I can't define this column as &quot;Gender&quot; in the code.</p>
  <p>Note: Here valid column means the columns which are determined by metadata/schema and invalid columns/extra columns can be defined as which are not present in metadata.</p>