title: Spark Driver Memory and Executor Memory
body: <p>I am beginner to Spark and I am running my application to read 14KB data from text filed, do some transformations and actions(collect, collectAsMap) and save data to Database</p>
  
  <p>I am running it locally in my macbook with 16G memory, with 8 logical cores.</p>
  
  <p>Java Max heap is set at 12G.</p>
  
  <p>Here is the command I use to run the application.</p>
  
  <p>bin/spark-submit --class com.myapp.application --master local[*] --executor-memory 2G --driver-memory 4G /jars/application.jar</p>
  
  <p>I am getting the following warning </p>
  
  <blockquote>
    <p>2017-01-13 16:57:31.579 [Executor task launch worker-8hread] WARN org.apache.spark.storage.MemoryStore - Not enough space to cache
    rdd_57_0 in memory! (computed 26.4 MB so far)</p>
  </blockquote>
  
  <p>Can anyone guide me on what is going wrong here and how can I improve performance? Also how to optimize on suffle-spill ? Here is a view of the spill that happens in my local system</p>
  
  <p><a href="https://i.stack.imgur.com/u4wie.png" rel="noreferrer"><img src="https://i.stack.imgur.com/u4wie.png" alt="enter image description here"></a></p>