title: Spark add new column to dataframe with value from previous row
body: <p>I'm wondering how I can achieve the following in Spark (Pyspark)</p>
  
  <p>Initial Dataframe:</p>
  
  <pre><code>+--+---+
  |id|num|
  +--+---+
  |4 |9.0|
  +--+---+
  |3 |7.0|
  +--+---+
  |2 |3.0|
  +--+---+
  |1 |5.0|
  +--+---+
  </code></pre>
  
  <p>Resulting Dataframe:</p>
  
  <pre><code>+--+---+-------+
  |id|num|new_Col|
  +--+---+-------+
  |4 |9.0|  7.0  |
  +--+---+-------+
  |3 |7.0|  3.0  |
  +--+---+-------+
  |2 |3.0|  5.0  |
  +--+---+-------+
  </code></pre>
  
  <p>I manage to generally "append" new columns to a dataframe by using something like:
  <code>df.withColumn("new_Col", df.num * 10)</code></p>
  
  <p>However I have no idea on how I can achieve this "shift of rows" for the new column, so that the new column has the value of a field from the previous row (as shown in the example). I also couldn't find anything in the API documentation on how to access a certain row in a DF by index.</p>
  
  <p>Any help would be appreciated.</p>