title: Is Dataset.JoinWith the best way to join two datasets in Spark?
body: <p>I have two large datasets, </p>
  
  <p><code>val dataA : Dataset[TypeA]</code> and
  <code>val dataB: Dataset[TypeB]</code>, where both <code>TypeA</code> and <code>TypeB</code> extend <code>Serializable</code>.</p>
  
  <p>I want to join the two datasets on separate columns, so where <code>TypeA.ColumnA == TypeB.ColumnB</code>. Spark offers the function <a href="https://spark.apache.org/docs/2.0.1/api/java/org/apache/spark/sql/Dataset.html#joinWith(org.apache.spark.sql.Dataset,%20org.apache.spark.sql.Column)" rel="nofollow noreferrer"><code>JoinWith</code></a> on a dataset, which I think will do this properly, but the function is undocumented and marked as "experimental". </p>
  
  <p>The other approach I have looked at is to use PairRDDs instead of datasets, and join them using a common key (like it says in this stackoverlow post here: <a href="https://stackoverflow.com/questions/39822377/how-to-join-two-datasets-by-key-in-scala-spark">how to join two datasets by key in scala spark</a>).</p>
  
  <p>Is there a specifically better approach to joining two datasets, or using either JoinWith or PairRDDs the best way?</p>