title: Allocating memory to context when using spark shell
body: <p>When I start start spark shell using : </p>
  
  <pre><code>./bin/spark-shell --master spark://IP:7077 --executor-memory 4G
  </code></pre>
  
  <p>Then no memory is allocated to Spark : </p>
  
  <p><img src="https://i.stack.imgur.com/gPiVd.png" alt="enter image description here"></p>
  
  <p>If however if I just use default : </p>
  
  <pre><code>./bin/spark-shell --master spark://IP:7077
  </code></pre>
  
  <p>then memory is allocated : </p>
  
  <p><img src="https://i.stack.imgur.com/iPgQD.png" alt="enter image description here"></p>
  
  <blockquote>
    <p>How can I use max available memory in spark shell ? In this case
    845MB+845MB+2.8GB = 4.49GB</p>
  </blockquote>
  
  <p>Update : It appears Spark will just allocate to each node the max available memory of the node with the least amount of memory. So if I use : </p>
  
  <pre><code>./bin/spark-shell --master spark://IP:7077 --executor-memory 845M
  </code></pre>
  
  <p>then 2 nodes are fully allocated but node with 2.8GB is not fully allocated : </p>
  
  <p><img src="https://i.stack.imgur.com/yB963.png" alt="enter image description here"></p>
  
  <p>So question now becomes can Spark be configured so that each node uses it's max free memory ?</p>