title: Spark 2.0.2 Data Frame - how to add a new column consist of first two chars of an existing column?
body: <p>coming from Pandas background, I am struggling with Spark 2.0.2 (PySpark) on a simple operation.  How to add a new column to an existing Data Frame, which contains the first two character of an existing column.  Ideally what I want essentially is </p>
  
  <pre><code>df.withColumn("new_column", col('FirstName')[:2])
  </code></pre>
  
  <p>which obviously Spark 2.0.2 cannot understand.</p>
  
  <p>Data Source</p>
  
  <pre><code>df = sqlContext.createDataFrame([("Ranjeet", 4, "California", 2), ("Anthony", 5, "Hawaii", 3), ("Diana", 6, "Singapore", 4)], ["FirstName", "Sales", "State", "Cost"])
  df.show()
  </code></pre>
  
  <p>Attempt #1</p>
  
  <pre><code>from pyspark.sql.functions import col
  df.withColumn("new_column", col('FirstName').map(lambda x: x[0][:2]))
  </code></pre>
  
  <p>Error: Column object is not callable</p>
  
  <p>Attemp #2 - creates a new data frame containing the first two chars of FirstName but how do I merge it back to df?  </p>
  
  <pre><code>from pyspark.sql import Row
  row = Row("block_x")
  df.select('FirstName').rdd.map(lambda x: x[0][:2]).map(row).toDF().show()
  </code></pre>
  
  <p>Thank you for your time.  </p>
  
  <p>Looking for a simple and elegant solution.  May be I am missing something here.</p>