title: Write data from a dstream to parquet
body: <p>After consuming data from kinesis using pyspark, I have a dstream with entries like this:</p>
  
  <pre><code>('filename_1', [{'name': 'test'}, {'name': 'more'}, {'name': 'other'}])
  ('filename_2', [{'age': 15}, {'age': 25}])
  </code></pre>
  
  <p>What I want to do now is to write the second part of the tuple to the location identified by the first part of the tuple.</p>
  
  <p>At other places I did that by converting each list of dictionaries to a DataFrame using:</p>
  
  <pre><code>dataframe = sqlContext.createDataFrame(list_of_dicts)
  </code></pre>
  
  <p>and writing it with something like:</p>
  
  <pre><code>dataframe.write.parquet('filename')
  </code></pre>
  
  <p>My problem is now how to turn every line in the dstream into a DataFrame. My intuition was to use map to get every single line and do the conversion. This would need a sqlContext which you can actually not pass to a map function, as it fails with this error:</p>
  
  <pre><code>Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063
  </code></pre>
  
  <p>I'm not absolutely tied to parquet, but I need some kind of schema (hence the detour to DataFrame). Is there a way to do that with spark?</p>