title: Remove duplicates from a dataframe in PySpark
body: <p>I'm messing around with dataframes in pyspark 1.4 locally and am having issues getting the <code>dropDuplicates</code> method to work. It keeps returning the error:</p>
  <blockquote>
  <p>&quot;AttributeError: 'list' object has no attribute 'dropDuplicates'&quot;</p>
  </blockquote>
  <p>Not quite sure why as I seem to be following the syntax in the <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame" rel="nofollow noreferrer">latest documentation</a>.</p>
  <pre><code>#loading the CSV file into an RDD in order to start working with the data
  rdd1 = sc.textFile(&quot;C:\myfilename.csv&quot;).map(lambda line: (line.split(&quot;,&quot;)[0], line.split(&quot;,&quot;)[1], line.split(&quot;,&quot;)[2], line.split(&quot;,&quot;)[3])).collect()
  
  #loading the RDD object into a dataframe and assigning column names
  df1 = sqlContext.createDataFrame(rdd1, ['column1', 'column2', 'column3', 'column4']).collect()
  
  #dropping duplicates from the dataframe
  df1.dropDuplicates().show()
  </code></pre>