title: Spark: Incremental collect() to a partition causes OutOfMemory in Heap
body: <p>I have a following code. Essentially I need to print RDD to a console, so I am collecting large RDD in smaller chunks by collecting it per partition. This is to avoid collecting entire RDD at once. When monitoring the heap and GC log, it seems like nothing is ever being GC'd. Heap keeps on growing until it hits OutOfMemory error. If my understanding is correct, in below once println statement is executed for collected RDDs, they won't be needed so it is safe to GC, but that's not what I see in the GC log each call to collect accumulates until OOM. Does anyone know why collected data is not being GC'd?</p>
  
  <pre><code> val writes = partitions.foreach { partition =&gt;
        val rddPartition = rdds.mapPartitionsWithIndex ({ 
          case (index, data) =&gt; if (index == partition.index) data else Iterator[Words]()
        }, false).collect().toSeq
        val partialReport = Report(rddPartition, reportId, dateCreated)
        println(partialReport.name) 
      }
  </code></pre>