title: Converting a Spark's DataFrame column to List[String] in Scala
body: <p>I am working on <a href="http://grouplens.org/datasets/movielens/" rel="nofollow" title="Movie Lens">Movie Lens</a> data set. In one the the <code>csv</code> files, the data is structured as:</p>
  
  <p><code>movieId</code> <code>movieTitle</code> <code>genres</code></p>
  
  <p>and <code>genres</code> again is a list of <code>|</code> separated values, the field is nullable.</p>
  
  <p>I am trying to get a unique list of all the <code>genres</code> so that I can rearrange the data as following:</p>
  
  <p><code>movieId</code> <code>movieTitle</code> <code>genre1</code> <code>genre2</code> <code>...</code> <code>genreN</code></p>
  
  <p>and a row, which has <code>genre</code> as <code>genre1 | genre2</code> will look like:</p>
  
  <p><code>1</code> <code>Title1</code> <code>1</code> <code>1</code> <code>0</code> <code>...</code> <code>0</code> </p>
  
  <p>So far, I have been able to read the <code>csv</code> file using the following code:</p>
  
  <pre><code>val conf         = new SparkConf().setAppName(App.name).setMaster(App.sparkMaster)
  val context      = new SparkContext(conf)
  val sparkSession = SparkSession.builder()
                     .appName(App.name)
                     .config("header", "true")
                     .config(conf = conf)
                     .getOrCreate()
  
  val movieFrame: DataFrame = sparkSession.read.csv(moviesPath)
  </code></pre>
  
  <p>If I try something like:</p>
  
  <pre><code>movieFrame.rdd.map(row ⇒ row(2).asInstanceOf[String]).collect()
  </code></pre>
  
  <p>Then I get the following exception:</p>
  
  <pre><code>java.lang.ClassNotFoundException: com.github.babbupandey.ReadData$$anonfun$1
  </code></pre>
  
  <p>Then, in addition, I tried providing the schema explicitly using the following code:</p>
  
  <pre><code>val moviesSchema: StructType = StructType(Array(StructField("movieId", StringType, nullable = true),
                                                          StructField("title", StringType, nullable = true),
                                                          StructField("genres", StringType, nullable = true)))
  </code></pre>
  
  <p>and tried:</p>
  
  <pre><code>val movieFrame: DataFrame = sparkSession.read.schema(moviesSchema).csv(moviesPath)
  </code></pre>
  
  <p>and then I got the same exception.</p>
  
  <p>Is there any way in which I can the set of <code>genres</code> as a <code>List</code> or a <code>Set</code> so I can further massage the data into the desired format? Any help will be appreciated.</p>