title: Understanding resource allocation for spark jobs on mesos
body: <p>I'm working on a project in Spark, and recently switched from using Spark Standalone to Mesos for cluster management. I now find myself confused about how to allocate resources when submitting a job under the new system.</p>
  
  <p>In standalone mode, I was using something like this (following some recommendations from <a href="http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/" rel="noreferrer">this Cloudera blog post</a>:</p>
  
  <pre><code>/opt/spark/bin/spark-submit --executor-memory 16G --executor-cores 8 
      --total-executor-cores 240 myscript.py
  </code></pre>
  
  <p>This is on a cluster where each machine has 16 cores and ~32 GB RAM.</p>
  
  <p>What was nice about this is that I had nice control over the number of executors running and the resources allocated to each. In the example above, I knew I was getting 240/8=30 executors, each with 16GB of memory and 8 cores. Given the memory on each machine in the cluster, this would amount to no more than two executors running on each machine. If I wanted more executors, I could do something like </p>
  
  <pre><code>/opt/spark/bin/spark-submit --executor-memory 10G --executor-cores 5 
      --total-executor-cores 240 myscript.py
  </code></pre>
  
  <p>This would now give me 240/5=47 executors, each with 5 cores and 10GB memory, and would allow up to 3 executors per machine.</p>
  
  <p>But now that I'm on mesos, I'm getting a bit confused. First off, I'm running in coarse-grained mode to ensure I can fix and control my resource allocation (this is in the service of fairly complex model where we want to pre-allocate resources).</p>
  
  <p>Now, I can specify <code>--total-executor-cores</code> and <code>--executor-memory</code>, but the documentation tells me that <code>--exeuctor-cores</code> applies to Spark standalone and YARN only, which makes specifying the total number of executors and resources allocated to each difficult. Say I run this:</p>
  
  <pre><code>/opt/spark/bin/spark-submit --total-executor-cores 240 --executor-memory 16G --conf spark.mesos.coarse=true myscript.py
  </code></pre>
  
  <p>When I examine this job in the Mesos web UI, things start getting confusing. So, here are my questions:</p>
  
  <ol>
  <li><p>Terminology. The Web UI lists "frameworks", which I assume correspond to "jobs" in the standalone UI. But when I click on the detail for a given framework, it lists "tasks". But these can't be actual Spark tasks, right? As far as I can tell, "task" here must actually mean "executor" as far as Spark is concerned. This would be consistent with the UI saying my framework (job) has: 15 active tasks, 240 CPUs, and 264GB memory.</p>
  
  <p>264/15=17.6, which seems consistent with the 16GB memory per executor I specified (plus some overhead, I guess). Am I right on how I'm interpreting all this?</p></li>
  <li><p>Assuming yes, when I examine any of these "tasks" (executors) I see that each has 16 cores assigned. Given we have 16 cores per machine, this would seem to indicate I'm basically running one executor on each of 16 machines, and that each executor is getting the full 16 cores, but only 16 GB of RAM. (note that, even if I drop <code>--executor-memory</code> way down, to something like 4GB, mesos still just runs one executor per node, with 16 cores and 4GB RAM). But what I want to accomplish is something like my first two examples. That is, I want to run multiple executors per node, each sharing the RAM and cores of that node (i.e.a moderate number of cores pre executor, 5-8). Considering I can't specify <code>--executor-cores</code> in Mesos, how do I accomplish this? Or am I way off base for some reason in even wanting to accomplish this? Will Mesos just not permit multiple exeuctors per node?</p></li>
  </ol>