title: Apache Spark: cache and partitions
body: <p>Here's situation: I have a constantly growing collection of data, which I want to process using RDD across the Hadoop cluster.</p>
  
  <p>Here is the short example:</p>
  
  <pre><code>val elementA = (1, Seq(2, 3))
  val elementB = (2, Seq(1, 3))
  val elementC = (3, Seq(1, 2))
  
  val testRDD = sc.parallelize(Seq(elementA, elementB, elementC)).
      map(x =&gt; (x._1, x._2)).setName("testRDD").cache()
  
  val elementD = (4, Seq(1, 3))
  val elementD1 = (1, Seq(4))
  val elementD2 = (3, Seq(4))
  
  val testAdd = sc.parallelize(Seq(elementD, elementD1, elementD2)).
      map(x =&gt; (x._1, x._2)).setName("testAdd")
  
  val testResult = testRDD.cogroup(testAdd).mapValues(x =&gt; (x._1 ++ x._2).flatten)
  </code></pre>
  
  <p>The result will be like this (order of elements can vary):</p>
  
  <pre><code>(1, List(2, 3, 4))
  (2, List(1, 3))
  (3, List(1, 2, 4))
  (4, List(1, 3))
  </code></pre>
  
  <p>Here's my goals:</p>
  
  <ol>
  <li>I want to <code>.cache()</code> my RDD in cluster memory.</li>
  <li>I want to have the ability to add new elements to the existing RDD.</li>
  </ol>
  
  <p>Here's what I've figured out:</p>
  
  <ol>
  <li>Each partition in RDD caches separately and entirely (for example, I had the collection with 100 elements and 4 partitions, I called <code>.cache().collect()</code> and <code>cache().first()</code> and got 4 cached partitions in first case and 1 in second case).</li>
  <li>The result of <code>testRDD.cogroup(testAdd)</code> is new RDD, that could be cached again, and if we'll try to use <code>var testRDD</code> and call <code>testRDD = testRDD.cogroup(testAdd)</code>, we'll lose the link to the cached data.</li>
  <li>I know, that RDD is most suiltable for batch applications, and I have this here: the <code>Seq()</code> for each new element will be computed from the properties of another elements.</li>
  </ol>
  
  <blockquote>
    <p>Is there any way to modify current RDD without removing all of it's elements from cache?</p>
  </blockquote>
  
  <p>I though about making a kind of temporary storage and merging temporary storage with current storage after reaching some limit on temporary storage...</p>