title: Low cpu usage while running a spark job
body: <p>I am running a Spark job. I have 4 cores and worker memory set to 5G. Application master is on another machine in the same network, and does not host any workers. This is my code:</p>
  <pre><code>private void myClass() {
      // configuration of the spark context
      SparkConf conf = new SparkConf().setAppName(&quot;myWork&quot;).setMaster(&quot;spark://myHostIp:7077&quot;).set(&quot;spark.driver.allowMultipleContexts&quot;, &quot;true&quot;);
      // creation of the spark context in wich we will run the algorithm
      JavaSparkContext sc = new JavaSparkContext(conf);
  
      // algorithm
      for(int i = 0; i&lt;200; i++) {
          System.out.println(&quot;===============================================================&quot;);
          System.out.println(&quot;iteration : &quot; + i);
          System.out.println(&quot;===============================================================&quot;);
          ArrayList&lt;Boolean&gt; list = new ArrayList&lt;Boolean&gt;();
          for(int j = 0; j &lt; 1900; j++){
              list.add(true);
          }
          JavaRDD&lt;Ant&gt; ratings = sc.parallelize(list, 100)
                      .map(bool -&gt; new myObj())
                      .map(obj -&gt; this.setupObj(obj))
                      .map(obj -&gt; this.moveObj(obj))
                      .cache();
          int[] stuff = ratings
                      .map(obj -&gt; obj.getStuff())
                      .reduce((obj1,obj2)-&gt;this.mergeStuff(obj1,obj2));
          this.setStuff(tour);
  
          ArrayList&lt;TabObj&gt; tabObj = ratings
                      .map(obj -&gt; this.objToTabObjAsTab(obj))
                      .reduce((obj1,obj2)-&gt;this.mergeTabObj(obj1,obj2));
          ratings.unpersist(false);
  
          this.setTabObj(tabObj);
      }
  
      sc.close();
  }
  </code></pre>
  <p>When I start it, I can see progress on the Spark UI, but it is really slow (I have to set the parrallelize quite high, otherwise I have a timeout issue). I thought it was a CPU bottleneck, but the JVM CPU consumption is actually very low (most of the time it is 0%, sometime a bit more than 5%...).</p>
  <p>The JVM is using around 3G Of memory according to the monitor, with only 19M cached.</p>
  <p>The master host has 4 cores, and less memory (4G). That machine shows 100% CPU consumption (a full core) and I don't understand why it is that high... It just has to send partitions to the worker on the other machine, right?</p>
  <p>Why is CPU consumption low on the worker, and high on the master?</p>