title: Count number of duplicate rows in SPARKSQL
body: <p>I have requirement where i need to count number of duplicate rows in SparkSQL for Hive tables.</p>
  
  <pre><code>from pyspark import SparkContext, SparkConf
  from pyspark.sql import HiveContext
  from pyspark.sql.types import *
  from pyspark.sql import Row
  app_name="test"
  conf = SparkConf().setAppName(app_name)
  sc = SparkContext(conf=conf)
  sqlContext = HiveContext(sc)
  df = sqlContext.sql("select * from  DV_BDFRAWZPH_NOGBD_R000_SG.employee")
  </code></pre>
  
  <p>As of now i have hardcoded the table name, but it actually comes as parameter. That being said we don't know the number of columns or their names as well.In python pandas we have something like df.duplicated.sum() to count number of duplicate records. Do we have something like this here?</p>
  
  <pre><code>+---+---+---+
  | 1 | A | B |
  +---+---+---+
  | 1 | A | B |
  +---+---+---+
  | 2 | B | E |
  +---+---+---+
  | 2 | B | E |
  +---+---+---+
  | 3 | D | G |
  +---+---+---+
  | 4 | D | G |
  +---+---+---+
  </code></pre>
  
  <p>Here number of duplicate rows are 4. (for example)</p>