title: Spark Scala read custom file format to dataframe with schema
body: <p>I have an input file which looks much like csv but with custom header:</p>
  
  <pre><code>FIELDS-START
  field1
  field2
  field3
  FIELDS-END-DATA-START
  val1,2,3
  val2,4,5
  DATA-END
  </code></pre>
  
  <p>Task:
  To read data to a typed dataframe, schema is obtained dynamically, example for this specific file:</p>
  
  <pre><code>val schema = StructType(
    StructField("field1", StringType, true) ::
      StructField("field2", IntegerType, true) ::
      StructField("field3", IntegerType, true) :: Nil
  )    
  </code></pre>
  
  <p>So bacause of custom header I can't use spark csv reader. Other thing I tried:  </p>
  
  <pre><code>val file = spark.sparkContext.textFile(...)
  val data: RDD[List[String]] = file.filter(_.contains(",")).map(_.split(',').toList)
  
  val df: DataFrame = spark.sqlContext.createDataFrame(data.map(Row.fromSeq(_)), schema)
  </code></pre>
  
  <p>It fails with runtime exception </p>
  
  <blockquote>
    <p><em>java.lang.String is not a valid external type for schema of int</em> which is because <em>createDataFrame</em> doesn't do any casting. </p>
  </blockquote>
  
  <p><strong>NOTE: Schema is obtained at runtime</strong> </p>
  
  <p>Thanks in advance!</p>