title: Spark: Programmatically creating dataframe schema in scala
body: <p>I have a smallish dataset that will be the result of a Spark job. I am thinking about converting this dataset to a dataframe for convenience at the end of the job, but have struggled to correctly define the schema. The problem is the last field below (<code>topValues</code>); it is an ArrayBuffer of tuples -- keys and counts.</p>
  
  <pre><code>  val innerSchema =
      StructType(
        Array(
          StructField("value", StringType),
          StructField("count", LongType)
        )
      )
    val outputSchema =
      StructType(
        Array(
          StructField("name", StringType, nullable=false),
          StructField("index", IntegerType, nullable=false),
          StructField("count", LongType, nullable=false),
          StructField("empties", LongType, nullable=false),
          StructField("nulls", LongType, nullable=false),
          StructField("uniqueValues", LongType, nullable=false),
          StructField("mean", DoubleType),
          StructField("min", DoubleType),
          StructField("max", DoubleType),
          StructField("topValues", innerSchema)
        )
      )
  
    val result = stats.columnStats.map{ c =&gt;
      Row(c._2.name, c._1, c._2.count, c._2.empties, c._2.nulls, c._2.uniqueValues, c._2.mean, c._2.min, c._2.max, c._2.topValues.topN)
    }
  
    val rdd = sc.parallelize(result.toSeq)
  
    val outputDf = sqlContext.createDataFrame(rdd, outputSchema)
  
    outputDf.show()
  </code></pre>
  
  <p>The error I'm getting is a MatchError: <code>scala.MatchError: ArrayBuffer((10,2), (20,3), (8,1)) (of class scala.collection.mutable.ArrayBuffer)</code></p>
  
  <p>When I debug and inspect my objects, I'm seeing this:</p>
  
  <pre><code>rdd: ParallelCollectionRDD[2]
  rdd.data: "ArrayBuffer" size = 2
  rdd.data(0): [age,2,6,0,0,3,14.666666666666666,8.0,20.0,ArrayBuffer((10,2), (20,3), (8,1))]
  rdd.data(1): [gender,3,6,0,0,2,0.0,0.0,0.0,ArrayBuffer((M,4), (F,2))]
  </code></pre>
  
  <p>It seems to me that I've accurately described the ArrayBuffer of tuples in my innerSchema, but Spark disagrees.</p>
  
  <p>Any idea how I should be defining the schema?</p>