title: When to use Spark DataFrame/Dataset API and when to use plain RDD?
body: <p>Spark SQL DataFrame/Dataset execution engine has several extremely efficient time &amp; space optimizations (e.g. InternalRow &amp; expression codeGen). According to many documentations, it seems to be a better option than RDD for most distributed algorithms.</p>
  
  <p>However, I did some sourcecode research and am still not convinced. I have no doubt that InternalRow is much more compact and can save large amount of memory. But execution of algorithms may not be any faster saving predefined expressions. Namely, it is indicated in sourcecode of  <code>org.apache.spark.sql.catalyst.expressions.ScalaUDF</code>, that every user defined function does 3 things:</p>
  
  <ol>
  <li>convert catalyst type (used in InternalRow) to scala type (used in GenericRow).</li>
  <li>apply the function</li>
  <li>convert the result back from scala type to catalyst type</li>
  </ol>
  
  <p>Apparently this is even slower than just applying the function directly on RDD without any conversion. Can anyone confirm or deny my speculation by some real-case profiling and code analysis?</p>
  
  <p>Thank you so much for any suggestion or insight.</p>