title: Hadoop Spark: How to make distinct of elements in JavaRDD?
body: <p>I want to store distinct of some JavaRDD collection to a file in Spark?</p>
  
  <p>By using distinct() method of RDD, I couldn’t achieve the same.</p>
  
  <p>My guess is RDD treats each element as a individual instance. How can we achieve the distinct in this case.</p>
  
  <p>Following is code snippet, Can anyone please help?</p>
  
  <pre><code>public static void main(String[] args) {
      SparkConf conf = new SparkConf().setAppName("Xml Spark Demo");
      JavaSparkContext sc = new JavaSparkContext(conf);
  
      JavaSQLContext sqlContext = new org.apache.spark.sql.api.java.JavaSQLContext(
              sc);
  
  
      // Load a text file and convert each line to a JavaBean.
      JavaRDD&lt;String&gt; dataFromFile = sc.textFile(
              "/home/kedarnath/Rentals/inputData/temp-01.xml").map(
              new ParseAgentFromXml());
  
      //Need distinct values here
      dataFromFile.distinct().saveAsTextFile("/home/kedarnath/Rentals/testOutputDistinct.txt");
  
  }
  </code></pre>
  
  <p>Thanks in advance,</p>
  
  <p>~Kedar</p>