title: Spark Cassandra Read and Write to Spark Csv Taking Hours
body: <p>We are reading from a Cassandra table and writing the data into a Csv file. The reads are quite fast in other tables but for a particular table the reads are taking hours to complete. We are reading only a particular month's data from this table using filters.</p>
  <p>The concern table schema is as below:</p>
  <pre><code>col1 partition key stringtype
  col2 clustering key datetype
  col3
  col4
  col5
  </code></pre>
  <p>In spark we used a filter to read like</p>
  <pre><code>tablename.filter($&quot;col2&quot; === &quot;2021-07-01&quot;)
  </code></pre>
  <p>Writing the filtered data to Csv as below:</p>
  <pre><code>(tablename.repartition(500)
  .write
  .options(Map(&quot;header&quot;-\&gt;&quot;false&quot;,&quot;delimiter&quot;-\&gt;&quot;,&quot;, &quot;timestampFormat&quot; -\&gt; &quot;yyyy-MM-dd HH:mm:ss.SSSZ&quot;))
  .format(&quot;com.databricks.spark.csv&quot;)
  .save(&quot;/FolderName&quot;))
  </code></pre>
  <p>We see there are only 67 stages running to complete this entire write and it is mostly same number of stages of other tables too.</p>
  <pre><code>Stage 0:=========================\&gt;                              (30 + 30) / 67\]
  </code></pre>
  <p>The entire write operation for the fitered data takes approx 3+ hours to complete for 25 Million records.</p>
  <p>We tried increasing and decreasing repartition before write and didn't find any reduction in runtime. Could you please suggest a better way to reduce this write time.</p>