title: Spark DataFrame to RDD - graph.apply with GraphX
body: <p>I have a vertices and edges file in order to construct a graph using the <code>apply</code> method with <code>GraphX.</code></p>
  
  <p>My input files are in <code>json.gz</code> format and I am thus using the spark <code>sqlContext.read.json</code> function to import the data. </p>
  
  <pre><code>val vertices = sqlContext.read.json("vertices.json.gz")
  
  res3: org.apache.spark.sql.DataFrame = [index: bigint, point: array&lt;double&gt;, toid: string]
  </code></pre>
  
  <p>Using <code>vertices.show()</code> I can inspect the input. </p>
  
  <pre><code>+-----+--------------------+--------------------+
  |index|               point|                toid|
  +-----+--------------------+--------------------+
  |    1|[508180.748, 1953...|osgb4000000031043205|
  |    2|[508163.122, 1953...|osgb4000000031043206|
  |    3|[508172.075, 1953...|osgb4000000031043207|
  |    4|[508513.0, 196023.0]|osgb4000000031043208|
  |    5|[514358.399, 1503...|osgb4000000029797733|
  </code></pre>
  
  <p>I have a <code>edges</code> file that I follow the same process with.</p>
  
  <p>I now wish to construct a graph with <code>GraphX</code> using the <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.graphx.Graph$@apply[VD,ED](RDD[(VertexId,VD)],RDD[Edge[ED]],VD)(ClassTag[VD],ClassTag[ED]):Graph[VD,ED]" rel="nofollow">Graph.apply</a> option. I know that this method does not accept <code>DataFrame</code> but rather <code>RDD</code> inputs.</p>
  
  <p>In order to convert the files I have tried using the <code>.rdd</code> option resulting in:</p>
  
  <pre><code>val vert_rdd = vertices.rdd
  vert_rdd: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = MapPartitionsRDD[12] at rdd at &lt;console&gt;:36
  </code></pre>
  
  <p>However, this outputs a <code>.sql.Row</code> object rather than a standard <code>RDD</code> object. Thus, it is not accepted as an input when creating a Graph using <code>graph.apply</code>. </p>
  
  <p>Is there a more direct method for using <code>JSON</code> files to GraphX or is there a different way to convert from a <code>DataFrame</code> to a <code>RDD</code>?</p>
  
  <p>Edit.</p>
  
  <p>Ultimately, the vertices file must be converted to a <a href="https://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/graphx/VertexRDD.html" rel="nofollow">VertexRDD</a> class. Where the index takes the place of <code>VertexId</code> and the <code>point</code> tuple and <code>toid</code> string are attributes. </p>
  
  <p>The edge file must be converted to a <a href="https://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/graphx/EdgeRDD.html" rel="nofollow">EdgeRDD</a> class. </p>
  
  <p><code>Edges</code> file looks like this:</p>
  
  <pre><code>edges: org.apache.spark.sql.DataFrame = [index: bigint, length: double, nature: string, negativeNode: string, polyline: array&lt;double&gt;, positiveNode: string, term: string, toid: string]
  
  +-----+------------------+------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
  |index|            length|            nature|        negativeNode|            polyline|        positiveNode|                term|                toid|
  +-----+------------------+------------------+--------------------+--------------------+--------------------+--------------------+--------------------+
  |    1| 112.8275895775762|Single Carriageway|osgb4000000023183407|[492019.481, 1565...|osgb4000000023183409|Private Road - Re...|osgb4000000023296573|
  |    2|141.57731318733806|Single Carriageway|osgb4000000023763485|[492144.493, 1567...|osgb4000000023183408|Private Road - Re...|osgb4000000023296574|
  |    3|190.23352139011513|Single Carriageway|osgb4000000023183650|[492835.25, 15687...|osgb4000000023183652|Private Road - Re...|osgb4000000023296638|
  </code></pre>