title: What is the most effective way to get elements of RDD in spark
body: <p>I need to get values of two columns of a dataframe converted to RDD. </p>
  
  <p>The first solution I have thought is that</p>
  
  <ul>
  <li>First convert the RDD to List of Row <code>RDD.collect()</code> </li>
  <li>then for each element of List, get values by using <code>Row[i].getInt(column_index)</code></li>
  </ul>
  
  <p>this solution works fine with small and medium size of data. But in large one, I got <code>over memory</code>. </p>
  
  <p>My temporary solution is that I only create newRDD which contains only two Columns instead all columns. And then, apply my solution above, this may reduce most of needed memory.</p>
  
  <p>Current implementation is like this: </p>
  
  <pre><code>Row[] rows = sparkDataFrame.collect();
  for (int i = 0; i &lt; rows.length; i++) { //about 50 million rows
    int yTrue = rows[i].getInt(0);
    int yPredict = rows[i].getInt(1);
  }
  </code></pre>
  
  <p>Could you help me to improve my solution, or suggest me other solutions! </p>
  
  <p>Thanks!</p>
  
  <p>ps: I'm a new spark's user! </p>