title: org.apache.spark.SparkException: job aborted due to stage failure: Task 0 in stage 2.0
body: <p>I was trying to solve the below problem statement using spark-shell in cloudera VM but getting error. Tried to follow few suggestions raised in other tickets on this site but not following. As I am new to this big data world so any help will be appreciated. </p>
  
  <pre><code>org.apache.spark.SparkException: **Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.lang.NumberFormatException: For input string: "patientID" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)**
  </code></pre>
  
  <p><strong>Problem Statement:</strong></p>
  
  <p>You have been given below patient data in csv format,</p>
  
  <pre><code>patientID,name,dateOfBirth,lastVisitDate
  1001,Ah Teck,1991-12-31,2012-01-20
  1002,Kumar,2011-10-29,2012-09-20
  1003,Ali,2011-01-30,2012-10-21
  </code></pre>
  
  <p>Accomplish following activities.</p>
  
  <p>1 . Find all the patients whose lastVisitDate between current time and '2012-09-15'</p>
  
  <p>Error message with scala code written in Spark-shell</p>
  
  <pre><code>[cloudera@quickstart ~]$ spark-shell
  Setting default log level to "WARN".
  To adjust logging level use sc.setLogLevel(newLevel).
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/avro/avro-tools-1.7.6-cdh5.12.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
        /_/
  
  Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67)
  Type in expressions to have them evaluated.
  Type :help for more information.
  17/11/06 20:15:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  Spark context available as sc (master = local[*], app id = local-1510028148251).
  17/11/06 20:15:53 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
  SQL context available as sqlContext.
  
  scala&gt; val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@3de10177
  
  scala&gt; import sqlContext.implicits._
  import sqlContext.implicits._
  
  scala&gt; import org.apache.spark.sql._
  import org.apache.spark.sql._
  
  scala&gt; val patients = sc.textFile("sparksql3/patients.csv")
  patients: org.apache.spark.rdd.RDD[String] = sparksql3/patients.csv MapPartitionsRDD[1] at textFile at &lt;console&gt;:35
  
  scala&gt; patients.first()
  res0: String = 1001,Ah Teck,1999-12-31,2012-01-20
  
  scala&gt; case class Patient(patientId:Integer,name:String,dateOfBirth:String,lastVisitDate:String)
  defined class Patient
  
  scala&gt; val patRDD = patients.map(_.split(",")).map(p=&gt;Patient(p(0).toInt,p(1),p(2),p(3)))
  patRDD: org.apache.spark.rdd.RDD[Patient] = MapPartitionsRDD[4] at map at &lt;console&gt;:39
  
  scala&gt; patRDD.count()
  17/11/06 20:27:01 ERROR executor.Executor: Exception in task 0.0 in stage 1.0 (TID 1)
  java.lang.NumberFormatException: For input string: "patientID"
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      at java.lang.Integer.parseInt(Integer.java:492)
      at java.lang.Integer.parseInt(Integer.java:527)
      at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
      at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
      at $line35.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at $line35.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1635)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  17/11/06 20:27:02 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.lang.NumberFormatException: For input string: "patientID"
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      at java.lang.Integer.parseInt(Integer.java:492)
      at java.lang.Integer.parseInt(Integer.java:527)
      at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
      at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
      at $line35.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at $line35.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1635)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  
  17/11/06 20:27:02 ERROR scheduler.TaskSetManager: Task 0 in stage 1.0 failed 1 times; aborting job
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.lang.NumberFormatException: For input string: "patientID"
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      at java.lang.Integer.parseInt(Integer.java:492)
      at java.lang.Integer.parseInt(Integer.java:527)
      at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
      at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1635)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1668)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1627)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1616)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
      at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1862)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1875)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1888)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1959)
      at org.apache.spark.rdd.RDD.count(RDD.scala:1157)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:42)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:47)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:49)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:51)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:53)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:55)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:57)
      at $iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:59)
      at $iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:61)
      at $iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:63)
      at $iwC$$iwC.&lt;init&gt;(&lt;console&gt;:65)
      at $iwC.&lt;init&gt;(&lt;console&gt;:67)
      at &lt;init&gt;(&lt;console&gt;:69)
      at .&lt;init&gt;(&lt;console&gt;:73)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at .&lt;init&gt;(&lt;console&gt;:7)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at $print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)
      at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)
      at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)
      at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)
      at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)
      at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
      at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
      at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
      at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
      at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
      at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
      at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
      at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
      at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
      at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
      at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
      at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)
      at org.apache.spark.repl.Main$.main(Main.scala:35)
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.NumberFormatException: For input string: "patientID"
      at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      at java.lang.Integer.parseInt(Integer.java:492)
      at java.lang.Integer.parseInt(Integer.java:527)
      at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
      at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(&lt;console&gt;:39)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1635)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1157)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1888)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>