{"action":"created","issue":{"url":"https://api.github.com/repos/saurfang/spark-sas7bdat/issues/32","repository_url":"https://api.github.com/repos/saurfang/spark-sas7bdat","labels_url":"https://api.github.com/repos/saurfang/spark-sas7bdat/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/saurfang/spark-sas7bdat/issues/32/comments","events_url":"https://api.github.com/repos/saurfang/spark-sas7bdat/issues/32/events","html_url":"https://github.com/saurfang/spark-sas7bdat/issues/32","id":300675870,"node_id":"MDU6SXNzdWUzMDA2NzU4NzA=","number":32,"title":"Cannot read SAS File","user":{"login":"TangoTom","id":36891696,"node_id":"MDQ6VXNlcjM2ODkxNjk2","avatar_url":"https://avatars0.githubusercontent.com/u/36891696?v=4","gravatar_id":"","url":"https://api.github.com/users/TangoTom","html_url":"https://github.com/TangoTom","followers_url":"https://api.github.com/users/TangoTom/followers","following_url":"https://api.github.com/users/TangoTom/following{/other_user}","gists_url":"https://api.github.com/users/TangoTom/gists{/gist_id}","starred_url":"https://api.github.com/users/TangoTom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TangoTom/subscriptions","organizations_url":"https://api.github.com/users/TangoTom/orgs","repos_url":"https://api.github.com/users/TangoTom/repos","events_url":"https://api.github.com/users/TangoTom/events{/privacy}","received_events_url":"https://api.github.com/users/TangoTom/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-02-27T15:27:52Z","updated_at":"2018-09-05T17:36:34Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\nI have a compressed SAS file with a few hundred columns, around 3.000.000 observations and file size 600MB. Reading the file ends in an error (appended).\r\nPlease ask if I can provide futher information.\r\nAny help very appreciated.\r\n\r\nThanks in advance for help, Thomas\r\n\r\nscala> <some object>.write.format(\"orc\").saveAsTable(\"<some name>\")\r\n[Stage 3:>                                                          (0 + 0) / 5]18/02/26 15:44:38 ERROR Utils: Aborting task\r\njava.lang.reflect.InvocationTargetException\r\n        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at com.github.saurfang.sas.util.PrivateMethodCaller.apply(PrivateMethodExposer.scala:27)\r\n        at com.github.saurfang.sas.parso.SasFileParserWrapper.readNext(ParsoWrapper.scala:80)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$lzycompute$1(SasRecordReader.scala:125)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$1(SasRecordReader.scala:124)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:137)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:33)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)\r\n        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n        at java.util.ArrayList.rangeCheck(ArrayList.java:653)\r\n        at java.util.ArrayList.get(ArrayList.java:429)\r\n        at com.epam.parso.impl.SasFileParser.readNext(SasFileParser.java:493)\r\n        ... 31 more\r\n18/02/26 15:44:38 WARN FileOutputCommitter: Could not delete hdfs://lde0162t.de.top.com:8020/apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000002_0\r\n18/02/26 15:44:38 ERROR FileFormatWriter: Job job_20180226154438_0003 aborted.\r\n18/02/26 15:44:38 ERROR Executor: Exception in task 2.0 in stage 3.0 (TID 11)\r\norg.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at com.github.saurfang.sas.util.PrivateMethodCaller.apply(PrivateMethodExposer.scala:27)\r\n        at com.github.saurfang.sas.parso.SasFileParserWrapper.readNext(ParsoWrapper.scala:80)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$lzycompute$1(SasRecordReader.scala:125)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$1(SasRecordReader.scala:124)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:137)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:33)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)\r\n        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\nCaused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n        at java.util.ArrayList.rangeCheck(ArrayList.java:653)\r\n        at java.util.ArrayList.get(ArrayList.java:429)\r\n        at com.epam.parso.impl.SasFileParser.readNext(SasFileParser.java:493)\r\n        ... 31 more\r\n18/02/26 15:44:38 WARN TaskSetManager: Lost task 2.0 in stage 3.0 (TID 11, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at com.github.saurfang.sas.util.PrivateMethodCaller.apply(PrivateMethodExposer.scala:27)\r\n        at com.github.saurfang.sas.parso.SasFileParserWrapper.readNext(ParsoWrapper.scala:80)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$lzycompute$1(SasRecordReader.scala:125)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$1(SasRecordReader.scala:124)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:137)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:33)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)\r\n        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\nCaused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n        at java.util.ArrayList.rangeCheck(ArrayList.java:653)\r\n        at java.util.ArrayList.get(ArrayList.java:429)\r\n        at com.epam.parso.impl.SasFileParser.readNext(SasFileParser.java:493)\r\n        ... 31 more\r\n\r\n18/02/26 15:44:38 ERROR TaskSetManager: Task 2 in stage 3.0 failed 1 times; aborting job\r\n18/02/26 15:44:38 ERROR Utils: Aborting task\r\norg.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n18/02/26 15:44:38 ERROR FileFormatWriter: Aborting job null.\r\norg.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 3.0 failed 1 times, most recent failure: Lost task 2.0 in stage 3.0 (TID 11, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at com.github.saurfang.sas.util.PrivateMethodCaller.apply(PrivateMethodExposer.scala:27)\r\n        at com.github.saurfang.sas.parso.SasFileParserWrapper.readNext(ParsoWrapper.scala:80)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$lzycompute$1(SasRecordReader.scala:125)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$1(SasRecordReader.scala:124)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:137)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:33)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)\r\n        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\nCaused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n        at java.util.ArrayList.rangeCheck(ArrayList.java:653)\r\n        at java.util.ArrayList.get(ArrayList.java:429)\r\n        at com.epam.parso.impl.SasFileParser.readNext(SasFileParser.java:493)\r\n        ... 31 more\r\n\r\nDriver stacktrace:\r\n        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)\r\n        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)\r\n        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)\r\n        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\n        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\r\n        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)\r\n        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)\r\n        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)\r\n        at scala.Option.foreach(Option.scala:257)\r\n        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)\r\n        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)\r\n        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)\r\n        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)\r\n        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\r\n        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)\r\n        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1928)\r\n        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1941)\r\n        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1961)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:127)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)\r\n        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:121)\r\n        at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:101)\r\n        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)\r\n        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)\r\n        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)\r\n        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)\r\n        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)\r\n        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)\r\n        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)\r\n        at org.apache.spark.sql.execution.datasources.DataSource.writeInFileFormat(DataSource.scala:484)\r\n        at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:500)\r\n        at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:263)\r\n        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)\r\n        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)\r\n        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n        at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)\r\n        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)\r\n        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)\r\n        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)\r\n        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)\r\n        at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:404)\r\n        at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:358)\r\n        at $line40.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:32)\r\n        at $line40.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:37)\r\n        at $line40.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:39)\r\n        at $line40.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:41)\r\n        at $line40.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:43)\r\n        at $line40.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:45)\r\n        at $line40.$read$$iw$$iw$$iw$$iw.<init>(<console>:47)\r\n        at $line40.$read$$iw$$iw$$iw.<init>(<console>:49)\r\n        at $line40.$read$$iw$$iw.<init>(<console>:51)\r\n        at $line40.$read$$iw.<init>(<console>:53)\r\n        at $line40.$read.<init>(<console>:55)\r\n        at $line40.$read$.<init>(<console>:59)\r\n        at $line40.$read$.<clinit>(<console>)\r\n        at $line40.$eval$.$print$lzycompute(<console>:7)\r\n        at $line40.$eval$.$print(<console>:6)\r\n        at $line40.$eval.$print(<console>)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)\r\n        at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)\r\n        at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)\r\n        at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)\r\n        at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)\r\n        at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)\r\n        at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)\r\n        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)\r\n        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)\r\n        at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)\r\n        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)\r\n        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)\r\n        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:415)\r\n        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:923)\r\n        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)\r\n        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)\r\n        at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)\r\n        at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)\r\n        at org.apache.spark.repl.Main$.doMain(Main.scala:69)\r\n        at org.apache.spark.repl.Main$.main(Main.scala:52)\r\n        at org.apache.spark.repl.Main.main(Main.scala)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:751)\r\n        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)\r\n        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)\r\n        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)\r\n        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\nCaused by: org.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at com.github.saurfang.sas.util.PrivateMethodCaller.apply(PrivateMethodExposer.scala:27)\r\n        at com.github.saurfang.sas.parso.SasFileParserWrapper.readNext(ParsoWrapper.scala:80)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$lzycompute$1(SasRecordReader.scala:125)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$1(SasRecordReader.scala:124)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:137)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:33)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)\r\n        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\nCaused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n        at java.util.ArrayList.rangeCheck(ArrayList.java:653)\r\n        at java.util.ArrayList.get(ArrayList.java:429)\r\n        at com.epam.parso.impl.SasFileParser.readNext(SasFileParser.java:493)\r\n        ... 31 more\r\n18/02/26 15:44:38 ERROR Utils: Aborting task\r\norg.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n18/02/26 15:44:38 WARN FileOutputCommitter: Could not delete hdfs://lde0162t.de.top.com:8020/apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000004_0\r\n18/02/26 15:44:38 ERROR FileFormatWriter: Job job_20180226154438_0003 aborted.\r\n18/02/26 15:44:38 ERROR Executor: Exception in task 4.0 in stage 3.0 (TID 13)\r\norg.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: org.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\n18/02/26 15:44:38 ERROR Utils: Aborting task\r\norg.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n18/02/26 15:44:38 WARN FileOutputCommitter: Could not delete hdfs://lde0162t.de.top.com:8020/apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000003_0\r\n18/02/26 15:44:38 ERROR FileFormatWriter: Job job_20180226154438_0003 aborted.\r\n18/02/26 15:44:38 ERROR Executor: Exception in task 3.0 in stage 3.0 (TID 12)\r\norg.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: org.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\n18/02/26 15:44:38 WARN DFSClient: DataStreamer Exception\r\norg.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000000_0/part-00000-1a342208-9aa9-47e8-894e-4dc804177880.snappy.orc (inode 7389846): File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-802042230_1, pendingcreates: 1]\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3660)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3463)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3301)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3261)\r\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:850)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:504)\r\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)\r\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:422)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)\r\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)\r\n\r\n        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)\r\n        at org.apache.hadoop.ipc.Client.call(Client.java:1498)\r\n        at org.apache.hadoop.ipc.Client.call(Client.java:1398)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)\r\n        at com.sun.proxy.$Proxy15.addBlock(Unknown Source)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:459)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)\r\n        at com.sun.proxy.$Proxy16.addBlock(Unknown Source)\r\n        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1574)\r\n        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1369)\r\n        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:558)\r\n18/02/26 15:44:38 WARN FileOutputCommitter: Could not delete hdfs://lde0162t.de.top.com:8020/apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000000_0\r\n18/02/26 15:44:38 ERROR FileFormatWriter: Job job_20180226154438_0003 aborted.\r\n18/02/26 15:44:38 WARN Utils: Suppressing exception in catch: No lease on /apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000000_0/part-00000-1a342208-9aa9-47e8-894e-4dc804177880.snappy.orc (inode 7389846): File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-802042230_1, pendingcreates: 1]\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3660)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3463)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3301)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3261)\r\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:850)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:504)\r\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)\r\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:422)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)\r\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)\r\n\r\norg.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000000_0/part-00000-1a342208-9aa9-47e8-894e-4dc804177880.snappy.orc (inode 7389846): File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-802042230_1, pendingcreates: 1]\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3660)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3463)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3301)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3261)\r\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:850)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:504)\r\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)\r\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:422)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)\r\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)\r\n\r\n        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)\r\n        at org.apache.hadoop.ipc.Client.call(Client.java:1498)\r\n        at org.apache.hadoop.ipc.Client.call(Client.java:1398)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)\r\n        at com.sun.proxy.$Proxy15.addBlock(Unknown Source)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:459)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)\r\n        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)\r\n        at com.sun.proxy.$Proxy16.addBlock(Unknown Source)\r\n        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1574)\r\n        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1369)\r\n        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:558)\r\n18/02/26 15:44:38 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 9)\r\norg.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: org.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\n        Suppressed: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154438_0003_m_000000_0/part-00000-1a342208-9aa9-47e8-894e-4dc804177880.snappy.orc (inode 7389846): File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-802042230_1, pendingcreates: 1]\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3660)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3463)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3301)\r\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3261)\r\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:850)\r\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:504)\r\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)\r\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)\r\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:422)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)\r\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)\r\n\r\n                at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)\r\n                at org.apache.hadoop.ipc.Client.call(Client.java:1498)\r\n                at org.apache.hadoop.ipc.Client.call(Client.java:1398)\r\n                at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)\r\n                at com.sun.proxy.$Proxy15.addBlock(Unknown Source)\r\n                at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:459)\r\n                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n                at java.lang.reflect.Method.invoke(Method.java:498)\r\n                at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)\r\n                at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)\r\n                at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)\r\n                at com.sun.proxy.$Proxy16.addBlock(Unknown Source)\r\n                at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1574)\r\n                at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1369)\r\n                at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:558)\r\norg.apache.spark.SparkException: Job aborted.\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:147)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)\r\n  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:121)\r\n  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:101)\r\n  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)\r\n  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)\r\n  at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)\r\n  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)\r\n  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)\r\n  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)\r\n  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)\r\n  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)\r\n  at org.apache.spark.sql.execution.datasources.DataSource.writeInFileFormat(DataSource.scala:484)\r\n  at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:500)\r\n  at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:263)\r\n  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)\r\n  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)\r\n  at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)\r\n  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)\r\n  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)\r\n  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)\r\n  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)\r\n  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)\r\n  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)\r\n  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:404)\r\n  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:358)\r\n  ... 50 elided\r\nCaused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 3.0 failed 1 times, most recent failure: Lost task 2.0 in stage 3.0 (TID 11, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\r\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n        at java.lang.reflect.Method.invoke(Method.java:498)\r\n        at com.github.saurfang.sas.util.PrivateMethodCaller.apply(PrivateMethodExposer.scala:27)\r\n        at com.github.saurfang.sas.parso.SasFileParserWrapper.readNext(ParsoWrapper.scala:80)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$lzycompute$1(SasRecordReader.scala:125)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.readNext$1(SasRecordReader.scala:124)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:137)\r\n        at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:33)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)\r\n        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)\r\n        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\nCaused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n        at java.util.ArrayList.rangeCheck(ArrayList.java:653)\r\n        at java.util.ArrayList.get(ArrayList.java:429)\r\n        at com.epam.parso.impl.SasFileParser.readNext(SasFileParser.java:493)\r\n        ... 31 more\r\n\r\nDriver stacktrace:\r\n  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)\r\n  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)\r\n  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)\r\n  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\n  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\r\n  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)\r\n  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)\r\n  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)\r\n  at scala.Option.foreach(Option.scala:257)\r\n  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)\r\n  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)\r\n  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)\r\n  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)\r\n  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\r\n  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)\r\n  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1928)\r\n  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1941)\r\n  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1961)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:127)\r\n  ... 82 more\r\nCaused by: org.apache.spark.SparkException: Task failed while writing rows\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n  at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n  at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.reflect.InvocationTargetException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n  at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)\r\n  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n  at java.lang.reflect.Method.invoke(Method.java:498)\r\n  at com.github.saurfang.sas.util.PrivateMethodCaller.apply(PrivateMethodExposer.scala:27)\r\n  at com.github.saurfang.sas.parso.SasFileParserWrapper.readNext(ParsoWrapper.scala:80)\r\n  at com.github.saurfang.sas.mapred.SasRecordReader.readNext$lzycompute$1(SasRecordReader.scala:125)\r\n  at com.github.saurfang.sas.mapred.SasRecordReader.readNext$1(SasRecordReader.scala:124)\r\n  at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:137)\r\n  at com.github.saurfang.sas.mapred.SasRecordReader.next(SasRecordReader.scala:33)\r\n  at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)\r\n  at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)\r\n  at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)\r\n  at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\r\n  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n  ... 8 more\r\nCaused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0\r\n  at java.util.ArrayList.rangeCheck(ArrayList.java:653)\r\n  at java.util.ArrayList.get(ArrayList.java:429)\r\n  at com.epam.parso.impl.SasFileParser.readNext(SasFileParser.java:493)\r\n  ... 31 more\r\n\r\nscala> 18/02/26 15:44:39 ERROR Utils: Aborting task\r\norg.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n18/02/26 15:44:39 WARN FileOutputCommitter: Could not delete hdfs://lde0162t.de.top.com:8020/apps/hive/warehouse/bicc.db/tbdl_gdgdra_drtv/_temporary/0/_temporary/attempt_20180226154439_0003_m_000001_0\r\n18/02/26 15:44:39 ERROR FileFormatWriter: Job job_20180226154439_0003 aborted.\r\n18/02/26 15:44:39 ERROR Executor: Exception in task 1.0 in stage 3.0 (TID 10)\r\norg.apache.spark.SparkException: Task failed while writing rows\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:99)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: org.apache.spark.TaskKilledException\r\n        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\r\n        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)\r\n        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\r\n        ... 8 more\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/saurfang/spark-sas7bdat/issues/comments/418816379","html_url":"https://github.com/saurfang/spark-sas7bdat/issues/32#issuecomment-418816379","issue_url":"https://api.github.com/repos/saurfang/spark-sas7bdat/issues/32","id":418816379,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODgxNjM3OQ==","user":{"login":"nelson2005","id":189607,"node_id":"MDQ6VXNlcjE4OTYwNw==","avatar_url":"https://avatars0.githubusercontent.com/u/189607?v=4","gravatar_id":"","url":"https://api.github.com/users/nelson2005","html_url":"https://github.com/nelson2005","followers_url":"https://api.github.com/users/nelson2005/followers","following_url":"https://api.github.com/users/nelson2005/following{/other_user}","gists_url":"https://api.github.com/users/nelson2005/gists{/gist_id}","starred_url":"https://api.github.com/users/nelson2005/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nelson2005/subscriptions","organizations_url":"https://api.github.com/users/nelson2005/orgs","repos_url":"https://api.github.com/users/nelson2005/repos","events_url":"https://api.github.com/users/nelson2005/events{/privacy}","received_events_url":"https://api.github.com/users/nelson2005/received_events","type":"User","site_admin":false},"created_at":"2018-09-05T17:36:34Z","updated_at":"2018-09-05T17:36:34Z","author_association":"NONE","body":"I'm experiencing the same problem, only for specific datasets.  It's repeatable- that is, if I resave the sas7bdat (from SPDS) the same sas7bdat is unreadable each time."}}
{"actor":{"display_login":"nelson2005"}}