title: Recovery after driver failure by exception with spark-streaming
body: <p>We are currently working on a system using kafka, spark streaming, and Cassandra as DB. We are using checkpointing based on the content here [<a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing]" rel="nofollow">http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing]</a>. Inside the function used to create the Streaming context, we use createDirectStream to create our DStream and from this point, we execute several transformations and actions derived from call saveToCassandra on different RDDs</p>
  
  <p>We are running different tests to establish how the application should recover when a failure occurs. Some key points about our scenario are:</p>
  
  <ul>
  <li><p>We are testing with a fixed number of records in kafka (between 10 million and 20 million), that means, we consume from kafka once and the application brings all the records from kafka.</p></li>
  <li><p>We are executing the application in --deploy-mode 'client' inside one of the workers, that means that we stop and start the driver manually.</p></li>
  </ul>
  
  <p>We are not sure how to handle exceptions after DStreams were created, for example, if while writing to cassandra all nodes are dead, we get an exception that aborts the job, but after re-submitting the application, that job is not re-scheduled and the application keeps consuming from kafka getting multiple 'isEmpty' calls. </p>
  
  <p>We made a couple of tests using 'cache' on the repartitioned RDD (which didn't work after a failure different than just stopping and starting the driver), and changing the parameters "query.retry.count", "query.retry.delay" and "spark.task.maxFailures" without success, e.g., the job is aborted after x failed times.</p>
  
  <p>At this point we are confused on how should we use the checkpoint to re-schedule jobs after a failure.</p>