{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22288","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22288/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22288/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22288/events","html_url":"https://github.com/apache/spark/pull/22288","id":355700841,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEyMTY5MTU0","number":22288,"title":"[SPARK-22148][SPARK-15815][Scheduler] Acquire new executors to avoid hang because of blacklisting","user":{"login":"dhruve","id":7732317,"node_id":"MDQ6VXNlcjc3MzIzMTc=","avatar_url":"https://avatars1.githubusercontent.com/u/7732317?v=4","gravatar_id":"","url":"https://api.github.com/users/dhruve","html_url":"https://github.com/dhruve","followers_url":"https://api.github.com/users/dhruve/followers","following_url":"https://api.github.com/users/dhruve/following{/other_user}","gists_url":"https://api.github.com/users/dhruve/gists{/gist_id}","starred_url":"https://api.github.com/users/dhruve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhruve/subscriptions","organizations_url":"https://api.github.com/users/dhruve/orgs","repos_url":"https://api.github.com/users/dhruve/repos","events_url":"https://api.github.com/users/dhruve/events{/privacy}","received_events_url":"https://api.github.com/users/dhruve/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":26,"created_at":"2018-08-30T18:23:59Z","updated_at":"2018-10-19T00:05:39Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22288","html_url":"https://github.com/apache/spark/pull/22288","diff_url":"https://github.com/apache/spark/pull/22288.diff","patch_url":"https://github.com/apache/spark/pull/22288.patch"},"body":"## What changes were proposed in this pull request?\r\nEvery time a task is unschedulable because of the condition where no. of task failures < no. of executors available, we currently abort the taskSet - failing the job. This change tries to acquire new executors if dynamicAllocation is turned on so that we can complete the job successfully.\r\n\r\n## How was this patch tested?\r\n\r\nI performed some manual tests to check and validate the behavior. \r\n\r\n```scala\r\nval rdd = sc.parallelize(Seq(1 to 10), 3)\r\n\r\nimport org.apache.spark.TaskContext\r\n\r\nval mapped = rdd.mapPartitionsWithIndex ( (index, iterator) => { if (index == 2) { Thread.sleep(30 * 1000); val attemptNum = TaskContext.get.attemptNumber; if (attemptNum < 3) throw new Exception(\"Fail for blacklisting\")};  iterator.toList.map (x => x + \" -> \" + index).iterator } )\r\n\r\nmapped.collect\r\n```\r\n\r\nNote: I am putting up this PR as initial draft to review the approach. \r\n\r\nTodo List:\r\n- Add unit tests\r\n- Agree upon the conf name & value and update the docs \r\n\r\nWe can build on this approach further by:\r\n- Taking into account static allocation\r\n- Querying the RM to figure out if its a small cluster, then try to wait some more or abort immediately.\r\n- Try to distinguish between waiting for time while you acquire an executor and time for being unable to schedule a task.\r\n\r\nOpen to suggestions.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/431203256","html_url":"https://github.com/apache/spark/pull/22288#issuecomment-431203256","issue_url":"https://api.github.com/repos/apache/spark/issues/22288","id":431203256,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTIwMzI1Ng==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-10-19T00:05:39Z","updated_at":"2018-10-19T00:05:39Z","author_association":"NONE","body":"**[Test build #97556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97556/testReport)** for PR 22288 at commit [`d2af73d`](https://github.com/apache/spark/commit/d2af73d4bd10676169f4fddc9dd262c97c0a9967).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}