{"action":"created","issue":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/257","repository_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark","labels_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/257/labels{/name}","comments_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/257/comments","events_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/257/events","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/257","id":309526899,"number":257,"title":"AttributeError: 'NoneType' object has no attribute 'get'","user":{"login":"amantrac","id":6917753,"avatar_url":"https://avatars1.githubusercontent.com/u/6917753?v=4","gravatar_id":"","url":"https://api.github.com/users/amantrac","html_url":"https://github.com/amantrac","followers_url":"https://api.github.com/users/amantrac/followers","following_url":"https://api.github.com/users/amantrac/following{/other_user}","gists_url":"https://api.github.com/users/amantrac/gists{/gist_id}","starred_url":"https://api.github.com/users/amantrac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amantrac/subscriptions","organizations_url":"https://api.github.com/users/amantrac/orgs","repos_url":"https://api.github.com/users/amantrac/repos","events_url":"https://api.github.com/users/amantrac/events{/privacy}","received_events_url":"https://api.github.com/users/amantrac/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-03-28T20:51:11Z","updated_at":"2018-03-29T16:39:51Z","closed_at":null,"author_association":"NONE","body":"Hi, it seems this error is reported many times, but after checking the various issue, did not find anything that matches my use case.\r\nSo, I am working on cluster mode, on yarn. \r\nMy job is working fine for a while, and then crashes with the following error:\r\n\r\nJob aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 12, hdpdatanode25): org.apache.spark.api.python.PythonException: Traceback (most recent call last):\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 111, in main\r\n    process()\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 106, in process\r\n    serializer.dump_stream(func(split_index, iterator), outfile)\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 317, in func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 759, in func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py\", line 310, in _train\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/tfspark.zip/tensorflowonspark/TFSparkNode.py\", line 100, in _get_manager\r\n    logging.info(\"Connected to TFSparkNode.mgr on {0}, ppid={1}, state={2}\".format(host, ppid, str(TFSparkNode.mgr.get('state'))))\r\nAttributeError: 'NoneType' object has no attribute 'get'\r\n\r\n\tat org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)\r\n\tat org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)\r\n\tat org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)\r\n\tat org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)\r\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)\r\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:277)\r\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:89)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\nDriver stacktrace\r\n\r\n\r\nMy running command is as follow:\r\n${SPARK_HOME}/bin/spark-submit \\\r\n--master yarn \\\r\n--deploy-mode cluster \\\r\n--queue ${QUEUE} \\\r\n--num-executors 4 \\\r\n--executor-memory 10G \\\r\n--total-executor-cores 4 \\\r\n--py-files TensorFlowOnSpark/tfspark.zip,TensorFlowOnSpark/examples/criteo/spark/xxx_dist.py \\\r\n--conf spark.dynamicAllocation.enabled=false \\\r\n--conf spark.yarn.maxAppAttempts=1 \\\r\n--archives hdfs:///user/${USER}/Python.zip#Python \\\r\n--conf spark.executorEnv.LD_LIBRARY_PATH=\"$LIB_HDFS:$LIB_JVM\" \\\r\n--conf spark.executorEnv.HADOOP_HDFS_HOME=\"$HADOOP_HDFS_HOME\" \\\r\n--conf spark.executorEnv.CLASSPATH=\"$($HADOOP_HOME/bin/hadoop classpath --glob):${CLASSPATH}\" \\\r\nTensorFlowOnSpark/examples/criteo/spark/xxx_spark.py \\\r\n--mode train \\\r\n--data ${TRAINING_DATA} \\\r\n--validation ${VALIDATION_DATA} \\\r\n--steps 1000000 \\\r\n--model ${MODEL_OUTPUT}\r\n\r\n\r\nCopy/Paste from log:\r\n\r\nContainer: container_e82_1515444508016_3200860_01_000009 on hdpdatanode25_45454\r\n=================================================================================\r\nLogType:stderr\r\nLog Upload Time:Wed Mar 28 19:51:04 +0000 2018\r\nLogLength:41976\r\nLog Contents:\r\nSLF4J: Class path contains multiple SLF4J bindings.\r\nSLF4J: Found binding in [jar:file:/data2/hadoop/yarn/local/usercache/amantrach/filecache/4004/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.\r\n4.2.0-258.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\r\nSLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]\r\n18/03/28 19:40:43 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]\r\n18/03/28 19:40:44 INFO SecurityManager: Changing view acls to: yarn,amantrach\r\n18/03/28 19:40:44 INFO SecurityManager: Changing modify acls to: yarn,amantrach\r\n18/03/28 19:40:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, am\r\nantrach); users with modify permissions: Set(yarn, amantrach)\r\n18/03/28 19:40:44 INFO SecurityManager: Changing view acls to: yarn,amantrach\r\n18/03/28 19:40:44 INFO SecurityManager: Changing modify acls to: yarn,amantrach\r\n18/03/28 19:40:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, am\r\nantrach); users with modify permissions: Set(yarn, amantrach)\r\n18/03/28 19:40:45 INFO Slf4jLogger: Slf4jLogger started\r\n18/03/28 19:40:45 INFO Remoting: Starting remoting\r\n18/03/28 19:40:45 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@hdpdatanode25:37929]\r\n18/03/28 19:40:45 INFO Utils: Successfully started service 'sparkExecutorActorSystem' on port 37929.\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data8/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444\r\n508016_3200860/blockmgr-0c51ff7c-5796-4eba-9c82-fbab024a3496\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data7/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444\r\n508016_3200860/blockmgr-737f3482-ede7-4d9a-a2c9-2d7a22f13f96\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data9/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444\r\n508016_3200860/blockmgr-a617b78a-5c59-483e-8f07-961e311fab80\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data2/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444\r\n508016_3200860/blockmgr-97a31a14-bb28-4700-9361-3b018fa5e6bc\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444\r\n508016_3200860/blockmgr-40834467-7b08-4a36-ab92-06fa5c4aede8\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data10/hadoop/yarn/local/usercache/amantrach/appcache/application_151544\r\n4508016_3200860/blockmgr-3f10beb5-1fb0-417e-999f-886496fc75ad\r\n\r\n\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data6/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-c602423e-6b78-4a03-90ec-fe94fe8c83ac\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data1/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-7a36dd58-8b8a-4f1d-8fd0-b9a829bf3d09\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data5/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-2942c138-b7c1-4f9b-a87c-b0ce99e3896a\r\n18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-3c044739-e0ad-4de8-916d-46d4d11e9ecf\r\n18/03/28 19:40:45 INFO MemoryStore: MemoryStore started with capacity 7.0 GB\r\n18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@10.4.107.217:44573\r\n18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Successfully registered with driver\r\n18/03/28 19:40:45 INFO Executor: Starting executor ID 5 on host hdpdatanode25\r\n18/03/28 19:40:45 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52308.\r\n18/03/28 19:40:45 INFO NettyBlockTransferService: Server created on 52308\r\n18/03/28 19:40:45 INFO BlockManagerMaster: Trying to register BlockManager\r\n18/03/28 19:40:45 INFO BlockManagerMaster: Registered BlockManager\r\n18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Got assigned task 7\r\n18/03/28 19:40:45 INFO Executor: Running task 1.1 in stage 1.0 (TID 7)\r\n18/03/28 19:40:45 INFO TorrentBroadcast: Started reading broadcast variable 2\r\n18/03/28 19:40:45 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 17.5 KB, free 17.5 KB)\r\n18/03/28 19:40:45 INFO TorrentBroadcast: Reading broadcast variable 2 took 123 ms\r\n18/03/28 19:40:46 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 70.5 KB, free 88.0 KB)\r\n18/03/28 19:40:46 INFO HadoopRDD: Input split: hdfs://kandula/user/amantrach/criteo_dataset/training/part-r-00061.deflate:0+3530368287\r\n18/03/28 19:40:46 INFO TorrentBroadcast: Started reading broadcast variable 0\r\n18/03/28 19:40:46 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 28.0 KB, free 115.9 KB)\r\n18/03/28 19:40:46 INFO TorrentBroadcast: Reading broadcast variable 0 took 17 ms\r\n18/03/28 19:40:46 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 343.1 KB, free 459.0 KB)\r\n18/03/28 19:40:46 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id\r\n18/03/28 19:40:46 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id\r\n18/03/28 19:40:46 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap\r\n18/03/28 19:40:46 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition\r\n18/03/28 19:40:46 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id\r\n18/03/28 19:40:46 INFO ZlibFactory: Successfully loaded & initialized native-zlib library\r\n18/03/28 19:40:46 INFO CodecPool: Got brand-new decompressor [.deflate]\r\n18/03/28 19:40:48 ERROR Executor: Exception in task 1.1 in stage 1.0 (TID 7)\r\norg.apache.spark.api.python.PythonException: Traceback (most recent call last):\r\n\r\n File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 111, in main\r\n    process()\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 106, in process\r\n    serializer.dump_stream(func(split_index, iterator), outfile)\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 317, in func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 759, in func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py\", line 310, in _train\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/tfspark.zip/tensorflowonspark/TFSparkNode.py\", line 100, in _get_manager\r\n    logging.info(\"Connected to TFSparkNode.mgr on {0}, ppid={1}, state={2}\".format(host, ppid, str(TFSparkNode.mgr.get('state'))))\r\nAttributeError: 'NoneType' object has no attribute 'get'\r\n\r\n        at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)\r\n        at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)\r\n        at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)\r\n        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)\r\n        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)\r\n        at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:89)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n18/03/28 19:40:48 INFO CoarseGrainedExecutorBackend: Got assigned task 9\r\n18/03/28 19:40:48 INFO Executor: Running task 0.1 in stage 1.0 (TID 9)\r\n18/03/28 19:40:48 INFO HadoopRDD: Input split: hdfs://kandula/user/amantrach/criteo_dataset/training/part-r-00060.deflate:0+3531024796\r\n18/03/28 19:40:48 ERROR Executor: Exception in task 0.1 in stage 1.0 (TID 9)\r\norg.apache.spark.api.python.PythonException: Traceback (most recent call last):\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 111, in main\r\n    process()\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 106, in process\r\n    serializer.dump_stream(func(split_index, iterator), outfile)\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 317, in func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 759, in func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py\", line 310, in _train\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/tfspark.zip/tensorflowonspark/TFSparkNode.py\", line 100, in _get_manager\r\n    logging.info(\"Connected to TFSparkNode.mgr on {0}, ppid={1}, state={2}\".format(host, ppid, str(TFSparkNode.mgr.get('state'))))\r\nAttributeError: 'NoneType' object has no attribute 'get'\r\n       at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)\r\n        at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)\r\n        at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)\r\n        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)\r\n        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)\r\n        at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)\r\n        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\r\n        at org.apache.spark.scheduler.Task.run(Task.scala:89)\r\n        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n18/03/28 19:40:48 INFO CoarseGrainedExecutorBackend: Got assigned task 10\r\n18/03/28 19:40:48 INFO Executor: Running task 1.2 in stage 1.0 (TID 10)\r\n18/03/28 19:40:48 INFO HadoopRDD: Input split: hdfs://kandula/user/amantrach/criteo_dataset/training/part-r-00061.deflate:0+3530368287\r\n18/03/28 19:40:48 ERROR Executor: Exception in task 1.2 in stage 1.0 (TID 10)\r\norg.apache.spark.api.python.PythonException: Traceback (most recent call last):\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 111, in main\r\n    process()\r\n  File \"/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py\", line 106, in process\r\n    serializer.dump_stream(func(split_index, iterator), outfile)\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py\", line 2346, in pipeline_func\r\n  File \"/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001:\r\n...\r\n\r\nAttached the full log of another run\r\n\r\nAlso, when checking the run, the system automatically tries to run new executors...which is a problem and should not happened with maxattempt fixed to 1, and dynamicallocation set to false.\r\n\r\nAny help is welcome,\r\nThanks\r\n\r\n[application_1515444508016_3202937.log](https://github.com/yahoo/TensorFlowOnSpark/files/1857737/application_1515444508016_3202937.log)\r\n"},"comment":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/comments/377296232","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/257#issuecomment-377296232","issue_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/257","id":377296232,"user":{"login":"leewyang","id":3676078,"avatar_url":"https://avatars2.githubusercontent.com/u/3676078?v=4","gravatar_id":"","url":"https://api.github.com/users/leewyang","html_url":"https://github.com/leewyang","followers_url":"https://api.github.com/users/leewyang/followers","following_url":"https://api.github.com/users/leewyang/following{/other_user}","gists_url":"https://api.github.com/users/leewyang/gists{/gist_id}","starred_url":"https://api.github.com/users/leewyang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leewyang/subscriptions","organizations_url":"https://api.github.com/users/leewyang/orgs","repos_url":"https://api.github.com/users/leewyang/repos","events_url":"https://api.github.com/users/leewyang/events{/privacy}","received_events_url":"https://api.github.com/users/leewyang/received_events","type":"User","site_admin":false},"created_at":"2018-03-29T16:39:51Z","updated_at":"2018-03-29T16:39:51Z","author_association":"CONTRIBUTOR","body":"By caching, I only meant [Spark RDD persistence](https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence), i.e. if your upstream preprocessing of data requires caching/persisting the RDD into Spark storage memory... If not (i.e. your just reading from disk), then you can tune down the [spark.memory.storageFraction](https://spark.apache.org/docs/latest/configuration.html), so less of your memory is \"reserved\" for RDD storage.\r\n\r\nAs for the queue, that was required to guarantee that the TF process (running in the background) could access the data being fed via Spark executor process (in the foreground).  A workaround would be to switch your code to InputMode.TENSORFLOW and use TF's built in `tf.data` APIs to read directly from HDFS."}}
{"actor":{"display_login":"leewyang"}}