title: Spark createDataFrame failing with ArrayOutOfBoundsException
body: <p>I'm pretty new to Spark and am having a problem converting an RDD to a DataFrame. What I'm trying to do is take a log file, convert it to JSON using an existing jar (returns a string), and then make that resulting json into a dataframe. Here is what I have so far:</p>
  
  <pre><code>val serverLog = sc.textFile("/Users/Downloads/file1.log")
  val jsonRows = serverLog.mapPartitions(partition =&gt; {
    val txfm = new JsonParser //*jar to parse logs to json*//
    partition.map(line =&gt; {
      Row(txfm.parseLine(line))
    })
  })
  </code></pre>
  
  <p>When I run a <code>take(2)</code> on this I get something like:</p>
  
  <pre><code>[{"pwh":"600","sVe":"10.0","psh":"667","udt":"mobile"}]
  [{"pwh":"800","sVe":"10.0","psh":"1000","udt":"desktop"}]
  </code></pre>
  
  <p>My problem comes here. I create a schema and try to create the df</p>
  
  <pre><code>val schema = StructType(Array(
    StructField("pwh",StringType,true),
    StructField("sVe",StringType,true),...))
  
  val jsonDf = sqlSession.createDataFrame(jsonRows, schema)
  </code></pre>
  
  <p>And the returned error is</p>
  
  <pre><code>java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: 1
  if (assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, pwh), StringType), true) AS _pwh#0
  +- if (assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object), 0, pwh), StringType), true)
  :- assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object).isNullAt
  :  :- assertnotnull(input[0, org.apache.spark.sql.Row, true], top level row object)
  :  :  +- input[0, org.apache.spark.sql.Row, true]
  :  +- 0
  :- null
  </code></pre>
  
  <p>Can someone tell me what I'm doing wrong here? Most of the SO answers I've found say I can use either <code>createDataFrame</code> or <code>toDF()</code>, but I've had no luck with either. I also tried converting the RDD to a <code>JavaRDD</code>, but that also did not work. Appreciate any insight you can give.</p>