title: Spark Caching: RDD Only 8% cached
body: <p>For my code snippet as below:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val levelsFile = sc.textFile(levelsFilePath)
  val levelsSplitedFile = levelsFile.map(line =&gt; line.split(fileDelimiter, -1))
  val levelPairRddtemp = levelsSplitedFile
                                .filter(linearr =&gt; ( linearr(pogIndex).length!=0))
                                .map(linearr =&gt; (linearr(pogIndex).toLong, levelsIndexes.map(x =&gt; linearr(x))
                                .filter(value =&gt; (!value.equalsIgnoreCase("") &amp;&amp; !value.equalsIgnoreCase(" ") &amp;&amp; !value.equalsIgnoreCase("null")))))
                                .mapValues(value =&gt; value.mkString(","))
                                .partitionBy(new HashPartitioner(24))
                                .persist(StorageLevel.MEMORY_ONLY_SER)
  
  levelPairRddtemp.count // just to trigger rdd creation
  </code></pre>
  
  <p><strong>Info</strong></p>
  
  <ol>
  <li>The size of the file is ~ 4G </li>
  <li>I am using 2 <code>executors</code>(5G each) and
  12 cores.</li>
  <li><code>Spark</code> version: 1.5.2</li>
  </ol>
  
  <p><strong>Problem</strong></p>
  
  <p>When I look at the <code>SparkUI</code> in <code>Storage tab</code>, What I see is :</p>
  
  <p><a href="https://i.stack.imgur.com/dL9eI.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dL9eI.jpg" alt="enter image description here"></a></p>
  
  <p>Looking inside the <code>RDD</code>, seems only 2 out of 24 <code>partitions</code> are cached.</p>
  
  <p><a href="https://i.stack.imgur.com/dUm5O.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dUm5O.jpg" alt="enter image description here"></a></p>
  
  <p>Any explanation to this behavior, and how to fix this. </p>
  
  <p><strong>EDIT 1</strong>: I just tried with 60 partitions for <code>HashPartitioner</code> as:</p>
  
  <pre><code>..
  .partitionBy(new HashPartitioner(60))
  ..
  </code></pre>
  
  <p>And it <strong>Worked</strong>. Now I am getting entire <code>RDD</code> cached. Any guess what might have happened here? Can data skewness cause this behavior?</p>
  
  <p><strong>Edit-2</strong>: Logs containing <code>BlockManagerInfo</code> when I ran again with 24 <code>partitions</code>. This time 3/24 <code>partitions</code> were cached:</p>
  
  <pre><code>16/03/17 14:15:28 INFO BlockManagerInfo: Added rdd_294_14 in memory on ip-10-1-34-66.ec2.internal:47526 (size: 107.3 MB, free: 2.6 GB) 
  16/03/17 14:15:30 INFO BlockManagerInfo: Added rdd_294_17 in memory on ip-10-1-34-65.ec2.internal:57300 (size: 107.3 MB, free: 2.6 GB) 
  16/03/17 14:15:30 INFO BlockManagerInfo: Added rdd_294_21 in memory on ip-10-1-34-65.ec2.internal:57300 (size: 107.4 MB, free: 2.5 GB)
  </code></pre>