title: Spark job running out of heap memory on takeSample
body: <p>I've got an Apache spark cluster with one master node and three worker nodes. The worker nodes have 32 cores and 124G of memory each.  I've also got a dataset in HDFS with around 650 million text records.  This dataset is a number of serialized RDDs read in like so:</p>
  
  <pre><code>import org.apache.spark.mllib.linalg.{Vector, Vectors, SparseVector}
  val vectors = sc.objectFile[(String, SparseVector)]("hdfs://mn:8020/data/*")
  </code></pre>
  
  <p>I'd like to extract a sample of one million of these records to do some analytics, so I figured I'd try <code>val sample = vectors.takeSample(false, 10000, 0)</code>.  However, that eventually fails with this error message:</p>
  
  <pre><code> 15/08/25 09:48:27 ERROR Utils: Uncaught exception in thread task-result-getter-3
  java.lang.OutOfMemoryError: Java heap space
          at org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:64)
          at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1239)
          at org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:61)
          at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
          at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
          at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:89)
          at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:79)
          at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51)
          at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
          at org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:50)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
  Exception in thread "task-result-getter-3" java.lang.OutOfMemoryError: Java heap space
          at org.apache.spark.scheduler.DirectTaskResult$$anonfun$readExternal$1.apply$mcV$sp(TaskResult.scala:64)
          at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1239)
          at org.apache.spark.scheduler.DirectTaskResult.readExternal(TaskResult.scala:61)
          at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
          at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
          at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:89)
          at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$r
  </code></pre>
  
  <p>I understand that I'm running out of heap space (on the driver, I think?), and that makes sense.  Doing <code>hadoop fs -du -s /path/to/data</code>, the dataset takes up 2575 Gigabytes on disk (but is only ~850 GB in size).</p>
  
  <p>So, my question is, what can I do to extract this sample of 1000000 records (which I later plan on serializing to disk)?  I know I could just do <code>takeSample()</code> with smaller sample sizes and aggregate them later, but I think I'm just not setting the correct configuration or doing something wrong, which is preventing me from doing this the way I'd like.</p>