title: How to load a csv directly into a Spark Dataset?
body: <p>I have a csv file [1] which I want to load directly into a Dataset. The problem is that I always get errors like</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Cannot up cast `probability` from string to float as it may truncate
  The type path of the target object is:
  - field (class: "scala.Float", name: "probability")
  - root class: "TFPredictionFormat"
  You can either add an explicit cast to the input data or choose a higher precision type of the field in the target object;
  </code></pre>
  
  <p>Moreover, and specifically for the <code>phrases</code> field (check case class [2]) it get</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: cannot resolve '`phrases`' due to data type mismatch: cannot cast StringType to ArrayType(StringType,true);
  </code></pre>
  
  <p>If I define all the fields in my case class [2] as type String then everything works fine but this is not what I want. Is there a simple way to do it [3]?</p>
  
  <hr>
  
  <p><strong>References</strong></p>
  
  <p>[1] An example row</p>
  
  <pre><code>B017NX63A2,Merrell,"['merrell_for_men', 'merrell_mens_shoes', 'merrel']",merrell_shoes,0.0806054356579781
  </code></pre>
  
  <p>[2] My code snippet is as follows</p>
  
  <pre><code>import spark.implicits._
  
  val INPUT_TF = "&lt;SOME_URI&gt;/my_file.csv"
  
  final case class TFFormat (
      doc_id: String,
      brand: String,
      phrases: Seq[String],
      prediction: String,
      probability: Float
  )
  
  val ds = sqlContext.read
  .option("header", "true")
  .option("charset", "UTF8")
  .csv(INPUT_TF)
  .as[TFFormat]
  
  ds.take(1).map(println)
  </code></pre>
  
  <p>[3] I have found ways to do it by first defining columns on a DataFrame level and the convert things to Dataset (like <a href="https://spark.apache.org/docs/1.6.1/sql-programming-guide.html#creating-datasets" rel="noreferrer">here</a> or <a href="https://stackoverflow.com/questions/29383107/how-to-change-column-types-in-spark-sqls-dataframe">here</a> or <a href="https://stackoverflow.com/questions/40080045/how-to-work-with-dataset-in-spark-using-scala">here</a>) but I am almost sure this is not the way things supposed to be done. I am also pretty sure that Encoders are probably the answer but I don't have a clue how</p>