{"action":"closed","issue":{"url":"https://api.github.com/repos/databricks/spark-avro/issues/139","repository_url":"https://api.github.com/repos/databricks/spark-avro","labels_url":"https://api.github.com/repos/databricks/spark-avro/issues/139/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-avro/issues/139/comments","events_url":"https://api.github.com/repos/databricks/spark-avro/issues/139/events","html_url":"https://github.com/databricks/spark-avro/issues/139","id":163184623,"number":139,"title":"Null pointer getting count from avro file  in mesos distributed","user":{"login":"joerwin","id":1033624,"avatar_url":"https://avatars.githubusercontent.com/u/1033624?v=3","gravatar_id":"","url":"https://api.github.com/users/joerwin","html_url":"https://github.com/joerwin","followers_url":"https://api.github.com/users/joerwin/followers","following_url":"https://api.github.com/users/joerwin/following{/other_user}","gists_url":"https://api.github.com/users/joerwin/gists{/gist_id}","starred_url":"https://api.github.com/users/joerwin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joerwin/subscriptions","organizations_url":"https://api.github.com/users/joerwin/orgs","repos_url":"https://api.github.com/users/joerwin/repos","events_url":"https://api.github.com/users/joerwin/events{/privacy}","received_events_url":"https://api.github.com/users/joerwin/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2016-06-30T14:49:44Z","updated_at":"2016-07-13T14:31:58Z","closed_at":"2016-07-13T14:31:58Z","body":"I have a need to utilize jersey 2 and process an avro file. I am trying to utilize spark 2.0.0-preview with com.databricks:spark-avro_2.10:3.0.0-preview for avro support. We are running the process within mesos. I have found that when the master is set to the mesos master that the process will fail with a null pointer exception, however when the master is set to local the process will succeed. I am creating a shadow jar that has org.scala-lang:scala-library::2.10.5, org.apache.hadoop:hadoop-aws:2.7.1 and com.databricks:spark-avro_2.10:3.0.0-preview packaged with the test class.\r\n\r\nThe sample code I am testing with:\r\n\r\n```scala\r\nimport org.apache.spark.sql.{DataFrame, SQLContext}\r\nimport org.apache.spark.sql.types.StructType\r\nimport org.apache.spark.{SparkConf, SparkContext}\r\nimport org.slf4j.{Logger, LoggerFactory}\r\n\r\n\r\nobject Test {\r\n\r\n  var schema: StructType = null\r\n  val logger: Logger  = LoggerFactory.getLogger(this.getClass)\r\n\r\n  def main(args: Array[String]): Unit = {\r\n    var fileType = args(0).toLowerCase\r\n    var fileLocation = args(1).toLowerCase\r\n    val (sqlContext) = setup()\r\n    run(sqlContext, fileType, fileLocation)\r\n  }\r\n\r\n  def setup() = {\r\n    val conf = new SparkConf().setAppName(\"TestSpark\")\r\n    conf.set( \"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\" )\r\n    conf.set(\"spark.sql.shuffle.partitions\", \"1\")\r\n    conf.set(\"fs.s3a.connection.maximum\",\"1500\")\r\n    conf.set(\"fs.s3a.threads.max\",\"2048\")\r\n    conf.set(\"fs.s3a.threads.core\",\"1500\")\r\n    conf.set(\"spark.sql.avro.compression.codec\",\"uncompressed\")\r\n\r\n    val sparkContext = new SparkContext(conf)\r\n    val sqlContext = new SQLContext(sparkContext)\r\n    sqlContext\r\n  }\r\n\r\n  def run(sqlContext: SQLContext, fileType: String, fileLocation: String) = {\r\n    if (fileType.equalsIgnoreCase(\"json\")) {\r\n      logger.info(\"Processing JSON File\")\r\n      countJson(sqlContext, fileLocation)\r\n    }\r\n    else if(fileType.equalsIgnoreCase(\"avro\")) {\r\n      logger.info(\"Processing AVRO File\")\r\n      countAvro(sqlContext, fileLocation)\r\n    }\r\n    else {\r\n      throw new IllegalArgumentException(\"Unkown File Format\");\r\n    }\r\n  }\r\n\r\n\r\n  def countAvro(sqlContext: SQLContext, fileLocation: String) {\r\n    var dataframe = sqlContext.read.format(\"com.databricks.spark.avro\").load(fileLocation).cache\r\n    logger.info(\"AVRO COUNT = \" + dataframe.count())\r\n  }\r\n\r\n  def countJson(sqlContext: SQLContext, fileLocation: String) {\r\n    var dataframe = sqlContext.read.json(fileLocation)\r\n    logger.info(\"JSON COUNT = \" + dataframe.count())\r\n  }\r\n}\r\n```\r\n\r\nOther Pertinent Details:\r\n\r\n- Mesos Version: 0.28.1\r\n- `$SPARK_HOME`: `/path/to/spark-2.0.0-preview-bin-hadoop2.7`\r\n\r\nUsing chronos to submit the jobs.\r\n\r\nI am unclear if this is a problem in the preview of spark or the preview of the avro support. Here are the findings of running avro and json counts in the mesos cluster.\r\n\r\n- Test 1: reading Avro file using mesos master\r\n\r\n  ```\r\n  $SPARK_HOME/bin/spark-submit\\\r\n    --conf spark.executor.uri=http://hosted/path/of/spark-2.0.0-preview-bin-hadoop2.7.tgz\\\r\n    --conf spark.app.name=TestSpark\\\r\n    --conf spark.executor.cores=1\\\r\n    --conf spark.cleaner.ttl=86400\\\r\n    --conf spark.worker.cleanup.enabled=true\\\r\n    --conf spark.worker.cleanup.appDataTtl=259200\\\r\n    --conf spark.local.dir=/tmp/\\\r\n    --deploy-mode client\\\r\n    --master mesos://zk://master.mesos:2181/mesos\\\r\n    --class Test testSpark.jar avro s3a://path/to//file.avro\r\n  ```\r\n\r\n  ```\r\n  FAILED:\r\n  Exception in thread \"main\" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-0-11-60.ec2.internal): java.lang.NullPointerException\r\n    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)\r\n    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)\r\n    at org.apache.avro.mapred.FsInput.<init>(FsInput.java:37)\r\n    at com.databricks.spark.avro.DefaultSource$$anonfun$buildReader$1.apply(DefaultSource.scala:146)\r\n    at com.databricks.spark.avro.DefaultSource$$anonfun$buildReader$1.apply(DefaultSource.scala:143)\r\n    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(fileSourceInterfaces.scala:278)\r\n    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(fileSourceInterfaces.scala:262)\r\n    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:114)\r\n    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:91)\r\n    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\r\n    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\r\n    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$7$$anon$1.hasNext(WholeStageCodegenExec.scala:357)\r\n    at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$3$$anon$1.hasNext(InMemoryTableScanExec.scala:178)\r\n    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:213)\r\n    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:911)\r\n    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:902)\r\n    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:858)\r\n    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:902)\r\n    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:660)\r\n    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:329)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:280)\r\n    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\r\n    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\r\n    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\r\n    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)\r\n    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)\r\n    at org.apache.spark.scheduler.Task.run(Task.scala:85)\r\n    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\r\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n    at java.lang.Thread.run(Thread.java:745)\r\n\r\n  Driver stacktrace:\r\n    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)\r\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)\r\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)\r\n    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\n    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\r\n    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)\r\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\r\n    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\r\n    at scala.Option.foreach(Option.scala:257)\r\n    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)\r\n    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)\r\n    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)\r\n    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)\r\n    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\r\n    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)\r\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1863)\r\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1876)\r\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1889)\r\n    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1903)\r\n    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:883)\r\n    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\r\n    at org.apache.spark.rdd.RDD.withScope(RDD.scala:357)\r\n    at org.apache.spark.rdd.RDD.collect(RDD.scala:882)\r\n    at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:290)\r\n    at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2122)\r\n    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)\r\n    at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2436)\r\n    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2121)\r\n    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2128)\r\n    at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2156)\r\n    at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2155)\r\n    at org.apache.spark.sql.Dataset.withCallback(Dataset.scala:2449)\r\n    at org.apache.spark.sql.Dataset.count(Dataset.scala:2155)\r\n    at Test$.countAvro(Test.scala:50)\r\n    at Test$.run(Test.scala:40)\r\n    at Test$.main(Test.scala:16)\r\n    at Test.main(Test.scala)\r\n    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n    at java.lang.reflect.Method.invoke(Method.java:498)\r\n    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724)\r\n    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)\r\n    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)\r\n    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)\r\n    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)\r\n  Caused by: java.lang.NullPointerException\r\n    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)\r\n    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)\r\n    at org.apache.avro.mapred.FsInput.<init>(FsInput.java:37)\r\n    at com.databricks.spark.avro.DefaultSource$$anonfun$buildReader$1.apply(DefaultSource.scala:146)\r\n    at com.databricks.spark.avro.DefaultSource$$anonfun$buildReader$1.apply(DefaultSource.scala:143)\r\n    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(fileSourceInterfaces.scala:278)\r\n    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(fileSourceInterfaces.scala:262)\r\n    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:114)\r\n    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:91)\r\n    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\r\n    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\r\n    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$7$$anon$1.hasNext(WholeStageCodegenExec.scala:357)\r\n    at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$3$$anon$1.hasNext(InMemoryTableScanExec.scala:178)\r\n    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:213)\r\n    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:911)\r\n    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:902)\r\n    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:858)\r\n    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:902)\r\n    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:660)\r\n    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:329)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:280)\r\n    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\r\n    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\r\n    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:318)\r\n    at org.apache.spark.rdd.RDD.iterator(RDD.scala:282)\r\n    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)\r\n    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)\r\n    at org.apache.spark.scheduler.Task.run(Task.scala:85)\r\n    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)\r\n    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n    at java.lang.Thread.run(Thread.java:745)\r\n  ```\r\n\r\n- Test 2: reading Avro file using local master\r\n\r\n  ```\r\n  $SPARK_HOME/bin/spark-submit\\\r\n    --conf spark.executor.uri=http://hosted/path/of/spark-2.0.0-preview-bin-hadoop2.7.tgz\\\r\n    --conf spark.app.name=TestSpark\\\r\n    --conf spark.executor.cores=1\\\r\n    --conf spark.cleaner.ttl=86400\\\r\n    --conf spark.worker.cleanup.enabled=true\\\r\n    --conf spark.worker.cleanup.appDataTtl=259200\\\r\n    --conf spark.local.dir=/tmp/\\\r\n    --deploy-mode client\\\r\n    --master local\\\r\n    --class Test testSpark.jar avro s3a://path/to//file.avro\r\n  ```\r\n\r\n  SUCCEEDED\r\n\r\n- Test 3: reading JSON file using mesos master\r\n\r\n  ```\r\n  $SPARK_HOME/bin/spark-submit\\\r\n    --conf spark.executor.uri=http://hosted/path/of/spark-2.0.0-preview-bin-hadoop2.7.tgz\\\r\n    --conf spark.app.name=TestSpark\\\r\n    --conf spark.executor.cores=1\\\r\n    --conf spark.cleaner.ttl=86400\\\r\n    --conf spark.worker.cleanup.enabled=true\\\r\n    --conf spark.worker.cleanup.appDataTtl=259200\\\r\n    --conf spark.local.dir=/tmp/\\\r\n    --deploy-mode client\\\r\n    --master mesos://zk://master.mesos:2181/mesos\\\r\n    --class Test testSpark.jar json s3a://path/to//file.json\r\n  ```\r\n\r\n  SUCCEEDED But had exception:\r\n\r\n  ```\r\n  org.apache.spark.SparkException: Exception thrown in awaitResult\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)\r\n    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)\r\n    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)\r\n    at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:414)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.executorTerminated(CoarseMesosSchedulerBackend.scala:553)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.statusUpdate(CoarseMesosSchedulerBackend.scala:494)\r\n  Caused by: org.apache.spark.SparkException: Could not find CoarseGrainedScheduler or it has been stopped.\r\n    at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:162)\r\n    at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:127)\r\n    at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:225)\r\n    at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:508)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)\r\n    ... 4 more\r\n  16/06/30 14:05:19 WARN NettyRpcEndpointRef: Error sending message [message = RemoveExecutor(15,Executor finished with state FINISHED)] in 2 attempts\r\n  org.apache.spark.SparkException: Exception thrown in awaitResult\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)\r\n    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)\r\n    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)\r\n    at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:414)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.executorTerminated(CoarseMesosSchedulerBackend.scala:553)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.statusUpdate(CoarseMesosSchedulerBackend.scala:494)\r\n  Caused by: org.apache.spark.SparkException: Could not find CoarseGrainedScheduler or it has been stopped.\r\n    at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:162)\r\n    at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:127)\r\n    at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:225)\r\n    at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:508)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)\r\n    ... 4 more\r\n  16/06/30 14:05:22 WARN NettyRpcEndpointRef: Error sending message [message = RemoveExecutor(15,Executor finished with state FINISHED)] in 3 attempts\r\n  org.apache.spark.SparkException: Exception thrown in awaitResult\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)\r\n    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)\r\n    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)\r\n    at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:414)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.executorTerminated(CoarseMesosSchedulerBackend.scala:553)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.statusUpdate(CoarseMesosSchedulerBackend.scala:494)\r\n  Caused by: org.apache.spark.SparkException: Could not find CoarseGrainedScheduler or it has been stopped.\r\n    at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:162)\r\n    at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:127)\r\n    at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:225)\r\n    at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:508)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)\r\n    ... 4 more\r\n  Exception in thread \"Thread-71\" org.apache.spark.SparkException: Error notifying standalone scheduler's driver endpoint\r\n    at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:417)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.executorTerminated(CoarseMesosSchedulerBackend.scala:553)\r\n    at org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.statusUpdate(CoarseMesosSchedulerBackend.scala:494)\r\n  Caused by: org.apache.spark.SparkException: Error sending message [message = RemoveExecutor(15,Executor finished with state FINISHED)]\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:119)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)\r\n    at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:414)\r\n    ... 2 more\r\n  Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)\r\n    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)\r\n    at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)\r\n    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)\r\n    ... 4 more\r\n  Caused by: org.apache.spark.SparkException: Could not find CoarseGrainedScheduler or it has been stopped.\r\n    at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:162)\r\n    at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:127)\r\n    at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:225)\r\n    at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:508)\r\n    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)\r\n    ... 4 more\r\n  ```\r\n\r\n- Test 4: reading JSON file using local master\r\n\r\n  ```\r\n  $SPARK_HOME/bin/spark-submit\\\r\n    --conf spark.executor.uri=http://hosted/path/of/spark-2.0.0-preview-bin-hadoop2.7.tgz\\\r\n    --conf spark.app.name=TestSpark\\\r\n    --conf spark.executor.cores=1\\\r\n    --conf spark.cleaner.ttl=86400\\\r\n    --conf spark.worker.cleanup.enabled=true\\\r\n    --conf spark.worker.cleanup.appDataTtl=259200\\\r\n    --conf spark.local.dir=/tmp/\\\r\n    --deploy-mode client\\\r\n    --master local\\\r\n    --class Test testSpark.jar json s3a://path/to//file.json\r\n  ```\r\n\r\n  SUCCEEDED\r\n"}}
{"actor":{"display_login":"joerwin"}}