title: How to generate lots of data in Spark?
body: <p>Currently when I want to generate data in Spark I do something like this:</p>
  
  <pre><code>//generates an array list of integers 0...999
  final List&lt;Integer&gt; range = range(1000);
  JavaRDD&lt;Data&gt; rdd = sc
     .parallelize(range)
     .mapPartitionsWithIndex(generateData(), false);
  </code></pre>
  
  <p>For large enough Range I run out of memory (500 million for example).</p>
  
  <p>How can get around this problem?</p>