title: Spark Streaming - java.io.IOException: Lease timeout of 0 seconds expired
body: <p>I have spark streaming application using checkpoint writing on HDFS.</p>
  
  <p>Has anyone know the solution? 
  Previously we were using the kinit to specify principal and keytab and got the suggestion to specify these via <code>spark-submit</code> command instead <code>kinit</code> but still this error and cause spark streaming application down.</p>
  
  <pre><code>spark-submit --principal sparkuser@HADOOP.ABC.COM --keytab /home/sparkuser/keytab/sparkuser.keytab --name MyStreamingApp --master yarn-cluster --conf "spark.driver.extraJavaOptions=-XX:+UseConcMarkSweepGC --conf "spark.eventLog.enabled=true" --conf "spark.streaming.backpressure.enabled=true" --conf "spark.streaming.stopGracefullyOnShutdown=true" --conf "spark.executor.extraJavaOptions=-XX:+UseConcMarkSweepGC --class com.abc.DataProcessor myapp.jar
  </code></pre>
  
  <p>I see multiple occurrences of following exception in logs and finally SIGTERM 15 that kills the executor and driver. We are using CDH 5.5.2</p>
  
  <blockquote>
    <p>2016-10-02 23:59:50 ERROR SparkListenerBus LiveListenerBus:96 -
    Listener EventLoggingListener threw an exception
    java.lang.reflect.InvocationTargetException
                   at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                   at java.lang.reflect.Method.invoke(Method.java:606)
                   at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:148)
                   at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:148)
                   at scala.Option.foreach(Option.scala:236)
                   at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:148)
                   at org.apache.spark.scheduler.EventLoggingListener.onUnpersistRDD(EventLoggingListener.scala:184)
                   at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:50)
                   at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
                   at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
                   at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56)
                   at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
                   at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79)
                   at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1135)
                   at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
    Caused by: java.io.IOException: Lease timeout of 0 seconds expired.
                   at org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2370)
                   at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:964)
                   at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:932)
                   at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
                   at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
                   at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
                   at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
                   at java.lang.Thread.run(Thread.java:745)</p>
  </blockquote>