title: Data Frame Mapping in spark scala?
body: <p>I have a data set like below </p>
  
  <pre><code>INo,Sc,Desc,Quan,IDate,UPice,CID,Country
  1,2,"H,H",6,12-1-2010 8:26,2.55,c1,United Kingdom
  2,3,"WE,RN",6,12-1-2010 8:26,3.39,c2,United Kingdom
  3,3,CREAM ,8,12-1-2010 8:26,2.75,c3,United Kingdom
  4,4,KLE,6,12-1-2010 8:26,3.39,c4,United Kingdom
  5,5,"DI,AMR,ROR, ",2,1-7-2011 13:55,11.02,c5,United Kingdom
  6,6,SU"asasa,1,12-9-2010 16:26,11.02,c6,United Kingdom
  </code></pre>
  
  <p>I have created Data frame by executing below code:</p>
  
  <pre><code>case class Rating(CID: String, SC: String, rating: Double)
  def parseRating(str: String): Rating = {
  val fields = str.split(",")
  Rating(fields(6), fields(1), 1.0D)
  }
  val ratings = spark.read.textFile("C:/Users/test/Desktop/test123.txt").map(parseRating).toDF().dropDuplicates("CID", "SC")
  ratings.collect().map(t =&gt; println(t))
  ratings.show()
  </code></pre>
  
  <p>I am getting output like below</p>
  
  <pre><code>+---------------+---+------+
  |            CID| SC|rating|
  +---------------+---+------+
  |12-9-2010 16:26|  6|   1.0|
  |             c3|  3|   1.0|
  |            CID| Sc|   1.0|
  |             c4|  4|   1.0|
  |           3.39|  3|   1.0|
  |              2|  5|   1.0|
  |           2.55|  2|   1.0|
  +---------------+---+------+
  </code></pre>
  
  <p>Here I came to because of <code>field=str.split(",")</code>.input data is
  6,6,"SET,CO,SU",1,12-9-2010 16:26,11.02,c6,United Kingdom</p>
  
  <p>But my expected output is </p>
  
  <pre><code>+---------------+---+------+
  |            CID| SC|rating|
  +---------------+---+------+
  |             c6|  6|   1.0|
  |             c3|  3|   1.0|
  |             c2|  3|   1.0|
  |             c4|  4|   1.0|
  |             c5|  5|   1.0|
  |             c1|  2|   1.0|
  +---------------+---+------+ 
  </code></pre>
  
  <p>Please help me on this.</p>