title: Explode in PySpark
body: <p>I would like to transform from a DataFrame that contains lists of words into a DataFrame with each word in its own row.</p>
  
  <p>How do I do explode on a column in a DataFrame?</p>
  
  <p>Here is an example with some of my attempts where you can uncomment each code line and get the error listed in the following comment. I use PySpark in Python 2.7 with Spark 1.6.1.</p>
  
  <pre><code>from pyspark.sql.functions import split, explode
  DF = sqlContext.createDataFrame([('cat \n\n elephant rat \n rat cat', )], ['word'])
  print 'Dataset:'
  DF.show()
  print '\n\n Trying to do explode: \n'
  DFsplit_explode = (
   DF
   .select(split(DF['word'], ' '))
  #  .select(explode(DF['word']))  # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;"
  #   .map(explode)  # AttributeError: 'PipelinedRDD' object has no attribute 'show'
  #   .explode()  # AttributeError: 'DataFrame' object has no attribute 'explode'
  ).show()
  
  # Trying without split
  print '\n\n Only explode: \n'
  
  DFsplit_explode = (
   DF 
   .select(explode(DF['word']))  # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;"
  ).show()
  </code></pre>
  
  <p>Please advice</p>