title: Spark(Standalone) didnot execute my job
body: <p>Alive Workers: 4
  Cores in use: 16 Total
  Memory in use: 27.2 GB Total</p>
  
  <p><strong>My driver program:</strong></p>
  
  <pre><code>if __name__ == '__main__':
  sc = SparkContext()
  
  
  image_list = []
  init_weight = 1 / (const.POSITIVE_SAMPLES_NUMBER * 2)
  sample_type = True
  
  acc_sample_count = sc.accumulator(-1)
  
  samp_list = []
  index = 0
  for image_name in os.listdir(const.POSITIVE_SAMPLES_PATH):
      image = Image.open(const.POSITIVE_SAMPLES_PATH+image_name)
      image_resize = image.resize((24, 24), Image.ANTIALIAS)
      image_gray_format = preoperate.format_gray(array(image_resize))
      image_list.append([index ,image_gray_format])
      index += 1
  
  image_rdd = sc.parallelize(image_list, 128).setName('image_rdd')
  
  samples_rdd = image_rdd.map(lambda img_index : generate_sample(img_index[1], sample_type, init_weight, img_index[0])).setName('sample_rdd')
  samples_list = samples_rdd.collect()
  print('#######################################')
  print(sys.getsizeof(samples_list))
  bc_samples = sc.broadcast(samples_list)
  del samples_list
  image_rdd.unpersist()
  y2_position_rdd = sc.parallelize(const.Y2_POSITION, 2000)
  y2_fea_val_rdd = y2_position_rdd.mapPartitions(lambda pos : map_cal(pos, bc_samples.value, haarcal.haar_like_Y2_cal), 2000).persist(StorageLevel.DISK_ONLY)
  take_list = y2_fea_val_rdd.take(1)
  print('#####################################')
  for sample in take_list[0][1]:
      print(sample.index)
  print('#####################################')
  print(y2_fea_val_rdd.count())
  </code></pre>
  
  <p>Here are two jobs:
  The first job it works well, but the secend job was ignored without any log.
  stderr in web ui without any err logs</p>
  
  <p><strong>My log:</strong></p>
  
  <pre><code>17/03/11 16:49:32 INFO TaskSetManager: Finished task 115.0 in stage 0.0 (TID 115) in 1463 ms on 10.29.90.41 (executor 3) (116/128)
  </code></pre>
  
  <p>17/03/11 16:49:32 INFO TaskSetManager: Finished task 116.0 in stage 0.0 (TID 116) in 1469 ms on 10.30.147.199 (executor 9) (117/128)</p>
  
  <p>17/03/11 16:49:32 INFO TaskSetManager: Finished task 117.0 in stage 0.0 (TID 117) in 1380 ms on 10.30.147.84 (executor 13) (118/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 119.0 in stage 0.0 (TID 119) in 1376 ms on 10.30.147.84 (executor 12) (119/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 118.0 in stage 0.0 (TID 118) in 1410 ms on 10.30.147.154 (executor 4) (120/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 120.0 in stage 0.0 (TID 120) in 1495 ms on 10.30.147.154 (executor 7) (121/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 122.0 in stage 0.0 (TID 122) in 1355 ms on 10.30.147.154 (executor 6) (122/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 123.0 in stage 0.0 (TID 123) in 1424 ms on 10.30.147.84 (executor 14) (123/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 121.0 in stage 0.0 (TID 121) in 1566 ms on 10.30.147.199 (executor 8) (124/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 124.0 in stage 0.0 (TID 124) in 1342 ms on 10.30.147.154 (executor 5) (125/128)</p>
  
  <p>17/03/11 16:49:33 INFO TaskSetManager: Finished task 125.0 in stage 0.0 (TID 125) in 1344 ms on 10.29.90.41 (executor 0) (126/128)</p>
  
  <p>17/03/11 16:49:34 INFO TaskSetManager: Finished task 126.0 in stage 0.0 (TID 126) in 1599 ms on 10.30.147.199 (executor 11) (127/128)</p>
  
  <p>17/03/11 16:49:35 INFO TaskSetManager: Finished task 127.0 in stage 0.0 (TID 127) in 2667 ms on 10.29.90.41 (executor 2) (128/128)</p>
  
  <p>17/03/11 16:49:35 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool</p>
  
  <p>17/03/11 16:49:35 INFO DAGScheduler: ResultStage 0 (collect at /home/dcooo/projects/FaceDetection/main/facedetection/spark_cal_features.py:70) finished in 16.419 s</p>
  
  <p>17/03/11 16:49:35 INFO DAGScheduler: Job 0 finished: collect at 
  /home/dcooo/projects/FaceDetection/main/facedetection/spark_cal_features.py:70, took 17.078753 s</p>
  
  #
  
  <p>40816</p>
  
  <p>17/03/11 16:49:37 INFO SparkContext: Invoking stop() from shutdown hook</p>
  
  <p>17/03/11 16:49:37 INFO SparkUI: Stopped Spark web UI at <a href="http://10.165.51.174:4040" rel="nofollow noreferrer">http://10.165.51.174:4040</a></p>
  
  <p>17/03/11 16:49:37 INFO StandaloneSchedulerBackend: Shutting down all executors</p>
  
  <p>17/03/11 16:49:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down</p>
  
  <p>17/03/11 16:49:37 INFO MapOutputTrackerMasterEndpoint: 
  MapOutputTrackerMasterEndpoint stopped!</p>
  
  <p>17/03/11 16:49:37 INFO MemoryStore: MemoryStore cleared</p>
  
  <p>17/03/11 16:49:37 INFO BlockManager: BlockManager stopped</p>
  
  <p>17/03/11 16:49:37 INFO BlockManagerMaster: BlockManagerMaster stopped</p>
  
  <p>17/03/11 16:49:37 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!</p>
  
  <p>17/03/11 16:49:37 INFO SparkContext: Successfully stopped SparkContext</p>
  
  <p>17/03/11 16:49:37 INFO ShutdownHookManager: Shutdown hook called</p>
  
  <p>17/03/11 16:49:37 INFO ShutdownHookManager: Deleting directory /tmp/spark-e8666f9e-c811-48f1-b116-7b48cd74347a</p>
  
  <p>17/03/11 16:49:37 INFO ShutdownHookManager: Deleting directory /tmp/spark-e8666f9e-c811-48f1-b116-7b48cd74347a/pyspark-a74f8c26-c7ef-43ed-ad5e-100a7e4c57e5</p>
  
  <p>My first job works well, and spark didnot execute my second job
  <strong>My environments:</strong></p>
  
  <pre><code>spark.executor.cores 1
  </code></pre>
  
  <p>spark.cores.max 16</p>
  
  <p>spark.executor.memory 1g</p>
  
  <p>spark.default.parallelism 128</p>