title: How to efficiently distinct count pyspark column(ids) which will add new ids with every run
body: <p>Scenario
  1st run/ Initial run - i have ids(1,2,1,3,4,5,6) - total distinct count of ids is 5 (1 week data)
  2nd run - i have ids(1,2,2,7,8,9) - total distinct count for both week is (1,2,3,4,5,6,7,8,9) i.e 9(both week data) - in order to get this i have created master data but count is not efficient.</p>
  <p>I am dealing with 3 months data and keeps on increasing every week, i need to perform distinct count till date(i.e 3months+1week...)</p>
  <p>Snippet:</p>
  <pre class="lang-py prettyprint-override"><code>df.write.mode('append').partitionBy('col').parquet(&quot;s3 loc for master data&quot;) // appending every weeks data to master data set
  </code></pre>
  <pre class="lang-py prettyprint-override"><code>df = spark.read.parquet(&quot;s3 loc for master data&quot;)
  df_count = df.select(&quot;id&quot;).distinct().count() 
  </code></pre>
  <p><code>distinct().count()</code> is taking time(since it will count existing plus new ids, i cannot skip including ids to avoid duplications)</p>
  <p>is there any way i can avoid counting same ids over and over again</p>