title: Write dataframe from spark cluster to cassandra cluster: Partitioning and Performance Tuning
body: <p>I have two clusters - 
  1. Cloudera Hadoop- Spark jobs run here
  2. Cloud - Cassandra cluster, multiple DC</p>
  
  <p>While writing a dataframe from my spark job to a cassandra cluster, I am doing a repartition (repartionCount=10) in spark before writing. See below:</p>
  
  <pre><code>import org.apache.spark.sql.cassandra._
  records.repartition(repartitionCount).write.cassandraFormat(table, keySpace)
    .mode(SaveMode.Append)
    .options(options)
    .option(CassandraConnectorConf.LocalDCParam.name, cassandraDC.name)
    .option(CassandraConnectorConf.ConnectionHostParam.name, cassandraDC.hosts)
    .save()
  </code></pre>
  
  <p>In my multi tenant spark cluster, for a spark batch load with 20M records, and below configs, I see lot of task failures, resource preemption and on the fly failures.</p>
  
  <pre><code>spark.cassandra.output.batch.grouping.buffer.size=1000
  spark.cassandra.output.batch.grouping.key=partition
  spark.cassandra.output.concurrent.writes=20 
  spark.cassandra.connection.compression=LZ4
  </code></pre>
  
  <p>How should I tune this? Is the repartition to blame?</p>
  
  <p>PS: My understanding in the beginning was: For a load with 20M rows, the "repartition" should distribute load evenly over executors (partition with 2M rows each), and batching will be done on these partition level (on 2M rows). But now, I am doubting is this causing unnecessary shuffle, if the spark-cassandra-connector is doing batching on whole dataframe level (whole 20M rows).</p>
  
  <p>UPDATE: Removing the "repartition" brought down the performance a lot on my cloudera spark cluster (default partitions set at spark level is - <code>spark.sql.shuffle.partitions: 200</code>), so i dug a bit deeper and found my initial understanding was correct. Please note my spark and cassandra clusters are different. Datastax spark-cassandra-connector opens one connection per partition with a cassandra coordinator node, so I have decided to let it be the same. As Alex suggested, I have reduced the concurrent writes, I believe that should help. </p>