title: PySpark - create multiple json files from dataframe
body: <p>I have data in the following format, which is obtained from Hive into a dataframe:</p>
  
  <pre><code>date, stock, price
  1388534400, GOOG, 50
  1388534400, FB, 60
  1388534400, MSFT, 55
  1388620800, GOOG, 52
  1388620800, FB, 61
  1388620800, MSFT, 55
  </code></pre>
  
  <p>Where date is the epoch for midnight on that day, and we have data going back 10 years or so (800million+ rows).
  My aim is to end up with a bunch of JSON files, one per stock, that would look like:</p>
  
  <pre><code>GOOG.json:
  {
  '1388534400': 50,
  '1388620800': 52
  }
  
  FB.json:
  {
  '1388534400': 60,
  '1388620800': 61
  }
  </code></pre>
  
  <p>A naive way would be to get a list of unique stocks and then get a subset of the dataframe by filtering out only those rows for each stock but this seems overly naive and horribly inefficient.
  Can this be done easily in Spark? I've currently got it working in native Python using PyHive, but due to the sheer volume of data, I'd rather have this done on a cluster/Spark.</p>