title: Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
body: <p>I have some data in the following format (either RDD or Spark DataFrame):</p>
  
  <pre><code>from pyspark.sql import SQLContext
  sqlContext = SQLContext(sc)
  
   rdd = sc.parallelize([('X01',41,'US',3),
                         ('X01',41,'UK',1),
                         ('X01',41,'CA',2),
                         ('X02',72,'US',4),
                         ('X02',72,'UK',6),
                         ('X02',72,'CA',7),
                         ('X02',72,'XX',8)])
  
  # convert to a Spark DataFrame                    
  schema = StructType([StructField('ID', StringType(), True),
                       StructField('Age', IntegerType(), True),
                       StructField('Country', StringType(), True),
                       StructField('Score', IntegerType(), True)])
  
  df = sqlContext.createDataFrame(rdd, schema)
  </code></pre>
  
  <p>What I would like to do is to 'reshape' the data, convert certain rows in Country(specifically US, UK and CA) into columns:</p>
  
  <pre><code>ID    Age  US  UK  CA  
  'X01'  41  3   1   2  
  'X02'  72  4   6   7   
  </code></pre>
  
  <p>Essentially, I need something along the lines of Python's <code>pivot</code> workflow:</p>
  
  <pre><code>categories = ['US', 'UK', 'CA']
  new_df = df[df['Country'].isin(categories)].pivot(index = 'ID', 
                                                    columns = 'Country',
                                                    values = 'Score')
  </code></pre>
  
  <p>My dataset is rather large so I can't really <code>collect()</code> and ingest the data into memory to do the reshaping in Python itself. Is there a way to convert Python's <code>.pivot()</code> into an invokable function while mapping either an RDD or a Spark DataFrame? Any help would be appreciated!</p>