title: pySpark withColumn with a function
body: <p>I have a dataframe which has 2 columns: <code>account_id</code> and <code>email_address</code>, now I want to add one more column <code>updated_email_address</code> which i call some function on <code>email_address</code> to get the <code>updated_email_address</code>. here is my code:</p>
  <pre><code>def update_email(email):
    print(&quot;== email to be updated: &quot; + email)
    today = datetime.date.today()
    updated = substring(email, -8, 8) + str(today.strftime('%m')) + str(today.strftime('%d')) + &quot;_updated&quot;
    return updated
  
  df.withColumn('updated_email_address', update_email(df.email_address))
  </code></pre>
  <p>but the result showed <code>updated_email_address</code> column as null:</p>
  <pre><code>+---------------+--------------+---------------------+
  |account_id     |email_address |updated_email_address|
  +---------------+--------------+---------------------+
  |123456gd7tuhha |abc@test.com  |null           |
  |djasevneuagsj1 |cde@test.com  |null           |
  +---------------+--------------+---------------+
  </code></pre>
  <p>inside the function <code>updated_email</code> it printed out:</p>
  <pre><code>Column&lt;b'(email_address + == email to be udpated: )'&gt;
  </code></pre>
  <p>also it showed the df's column data type as:</p>
  <pre><code>dfData:pyspark.sql.dataframe.DataFrame
  account_id:string
  email_address:string
  updated_email_address:double
  </code></pre>
  <p>why is <code>updated_email_address</code> column type of double?</p>