title: Using an if/then type statement in PySpark join
body: <p>I'm converting some old SAS code to PySpark and need to translated the below SAS line into the following PySpark join. Any hints? The below Pyspark code yields &quot;TypeError: condition should be a Column&quot;</p>
  <p>I need to make a new variable (VAR1_recode) and it needs to be null if VAR2 is also null. Otherwise it should have the same value as VAR1. VAR1 is numeric and VAR2 a string.</p>
  <pre><code>SAS:
  DATA NEED;
  MERGE HAVE1 HAVE2;
  BY ID;
  IF VAR2 = '' THEN VAR1 = '';
  RUN;
  </code></pre>
  <p>--</p>
  <pre><code>PYSPARK:
  import pyspark.sql.functions as F
  NEED = HAVE1.join(HAVE2, on=['ID'], how=&quot;left&quot;) \
              .withColumn('VAR1_recode',F.when(F.col('VAR2').isNull('')).otherwise(F.col('VAR1'))) \
              .select('ID',F.col('VAR1_recode').alias('VAR1'),'VAR2')
  </code></pre>