title: Why collecting a large dataset succeeds after increasing memoryOverhead
body: <h1>intro</h1>
  <p>In the apache spark documentation I see that the memory is divided into three groups, which can be configured using several parameters.
  Lets say we've got an AWS EMR machine m4.xlarge. On this machine Yarn has maximum allocation memory equal to 12288 MB. Using these configuration parameters:</p>
  <ul>
  <li>spark.(executor|driver).memoryOverhead = 0.2</li>
  <li>spark.(executor|driver).memory = 10g</li>
  <li>spark.memory.fraction = 0.6 (the default)</li>
  <li>spark.storage.fraction = 0.3 (the default)</li>
  </ul>
  <p>I get:</p>
  <ul>
  <li>memory overhead = 2G</li>
  <li>executor memory = 10G
  <ul>
  <li>execution memory = 3G (<code>spark.executor.memory * spark.memory.fraction * spark.storage.fraction</code>)</li>
  <li>storage memory = 3G (<code>spark.executor.memory * spark.memory.fraction * spark.storage.fraction</code>)</li>
  <li>user memory = 4G (<code>spark.executor.memory * 1-spark.memory.fraction</code>)</li>
  </ul>
  </li>
  </ul>
  <p>I'm using the same configuration for both driver and executor.</p>
  <p>First of all - is this calculation correct? Are these parameters OK? I'm mainly wondering whether it will leave enough RAM on the machine so that f.ex. YARN daemons won't fail?</p>
  <h1>main question</h1>
  <p>What is exactly stored in these memory areas?</p>
  <p>I'm wondering because I'm doing a fairly large collect (creating a ~1.5G Map[(Long, Long)]), which I then intend to broadcast to all executors. When I did the collect without explicitly specifying the overhead (default being 0.1) the cluster failed, containers were killed by YARN for exceeding memory limits, but with overhead at 0.2 everything goes smoothly. It seems as though my Map is stored in the overhead, but what is the purpose of the executor storage then?</p>
  <p>Thanks in advance!</p>