title: Spark job throwing "java.lang.OutOfMemoryError: GC overhead limit exceeded"
body: <p>I have a Spark job that throws "java.lang.OutOfMemoryError: GC overhead limit exceeded".</p>
  
  <p>The job is trying to process a filesize 4.5G.</p>
  
  <p>I've tried following spark configuration:</p>
  
  <pre><code>--num-executors 6  --executor-memory 6G --executor-cores 6 --driver-memory 3G 
  </code></pre>
  
  <p>I tried increasing more cores and executor which sometime works, but takes over 20 minutes to process the file.</p>
  
  <p>Could I do something to improve the performance? or stop the Java Heap issue?</p>