{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/72534813","id":72534813,"diff_hunk":"@@ -236,29 +245,42 @@ private[spark] class TaskSchedulerImpl(\n    * given TaskSetManager have completed, so state associated with the TaskSetManager should be\n    * cleaned up.\n    */\n-  def taskSetFinished(manager: TaskSetManager): Unit = synchronized {\n+  def taskSetFinished(manager: TaskSetManager, success: Boolean): Unit = synchronized {\n     taskSetsByStageIdAndAttempt.get(manager.taskSet.stageId).foreach { taskSetsForStage =>\n       taskSetsForStage -= manager.taskSet.stageAttemptId\n       if (taskSetsForStage.isEmpty) {\n         taskSetsByStageIdAndAttempt -= manager.taskSet.stageId\n       }\n     }\n     manager.parent.removeSchedulable(manager)\n-    logInfo(\"Removed TaskSet %s, whose tasks have all completed, from pool %s\"\n-      .format(manager.taskSet.id, manager.parent.name))\n+    if (success) {\n+      blacklistTracker.foreach(_.taskSetSucceeded(manager.execToFailures))\n+      logInfo(s\"Removed TaskSet ${manager.taskSet.id}, whose tasks have all completed, from pool\" +\n+        s\" ${manager.parent.name}\")\n+    } else {\n+      logInfo(s\"Removed TaskSet ${manager.taskSet.id}, since it failed, from pool\" +\n+        s\" ${manager.parent.name}\")\n+    }\n   }\n \n   private def resourceOfferSingleTaskSet(\n       taskSet: TaskSetManager,\n       maxLocality: TaskLocality,\n-      shuffledOffers: Seq[WorkerOffer],\n+      shuffledOffers: IndexedSeq[WorkerOffer],\n       availableCpus: Array[Int],\n       tasks: Seq[ArrayBuffer[TaskDescription]]) : Boolean = {\n     var launchedTask = false\n+    // nodes and executors that are blacklisted for the entire application have already been\n+    // filtered out by this point\n     for (i <- 0 until shuffledOffers.size) {\n-      val execId = shuffledOffers(i).executorId\n-      val host = shuffledOffers(i).host\n-      if (availableCpus(i) >= CPUS_PER_TASK) {\n+      val offer = shuffledOffers(i)\n+      val host = offer.host\n+      val execId = offer.executorId\n+      val offerBlacklisted = blacklistTracker.map { bl =>\n+        taskSet.isNodeBlacklistedForTaskSet(host) ||\n+          taskSet.isExecutorBlacklistedForTaskSet(execId)\n+      }.getOrElse(false)\n+      if (!offerBlacklisted && availableCpus(i) >= CPUS_PER_TASK) {","path":"core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala","position":102,"original_position":102,"commit_id":"c9e3662dd468fe262fcb663ca1898cac0029f8d7","original_commit_id":"c9e3662dd468fe262fcb663ca1898cac0029f8d7","user":{"login":"kayousterhout","id":1108612,"avatar_url":"https://avatars.githubusercontent.com/u/1108612?v=3","gravatar_id":"","url":"https://api.github.com/users/kayousterhout","html_url":"https://github.com/kayousterhout","followers_url":"https://api.github.com/users/kayousterhout/followers","following_url":"https://api.github.com/users/kayousterhout/following{/other_user}","gists_url":"https://api.github.com/users/kayousterhout/gists{/gist_id}","starred_url":"https://api.github.com/users/kayousterhout/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kayousterhout/subscriptions","organizations_url":"https://api.github.com/users/kayousterhout/orgs","repos_url":"https://api.github.com/users/kayousterhout/repos","events_url":"https://api.github.com/users/kayousterhout/events{/privacy}","received_events_url":"https://api.github.com/users/kayousterhout/received_events","type":"User","site_admin":false},"body":"It seems more intuitive to me to move all of the taskset-specific blacklisting to the TaskSetManager, since in general that class encompasses all of the taskset-specific logic.  E.g., taskSet.abortIfCompletelyBlacklisted could itself check with the blacklist tracker.  That would also allow the TaskSetMAnager to hide some complexity, like checking if the executor is blacklisted for the task set.\r\n\r\nCalling taskSetSucceeded could also move to the TaskSetManager, so that \"success\" didn't need to be passed into taskSetFinished.","created_at":"2016-07-27T22:18:33Z","updated_at":"2016-07-27T22:18:33Z","html_url":"https://github.com/apache/spark/pull/14079#discussion_r72534813","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/14079","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/72534813"},"html":{"href":"https://github.com/apache/spark/pull/14079#discussion_r72534813"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/14079"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14079","id":76512332,"html_url":"https://github.com/apache/spark/pull/14079","diff_url":"https://github.com/apache/spark/pull/14079.diff","patch_url":"https://github.com/apache/spark/pull/14079.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/14079","number":14079,"state":"open","locked":false,"title":"[SPARK-8425][CORE] New Blacklist Mechanism","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis adds a blacklisting mechanism for Spark to handle bad executors and nodes, eg. for when a disk fails.  Previously there was an undocumented blacklisting mechanism, but it was very limited.  This supports blacklisting individual tasks on an executor, executors and nodes for an entire stage, and executors and nodes across an entire spark application.  Full details are in a design doc attached to the jira.\r\n\r\n## How was this patch tested?\r\n\r\nAdded unit tests, ran them via Jenkins, also ran a handful of them in a loop to check for flakiness.\r\n\r\nThe added tests include:\r\n* verifying BlacklistTracker works correctly\r\n* verifying TaskSchedulerImpl interacts with BlacklistTracker correctly (via a mock BlacklistTracker)\r\n* an integration test for the entire scheduler with blacklisting in a few different scenarios\r\n\r\nTODO\r\n- [x] More testing on a real cluster\r\n- [x] Fix & test handling of shuffle-fetch failures\r\n- [x] Fix & test single task failure on a bad node should not trigger job failure\r\n- [x] Documentation","created_at":"2016-07-06T22:26:00Z","updated_at":"2016-07-27T22:18:33Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1bb14cb9ce0b73a303d6776cb3aabe36ccc325e0","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/14079/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/14079/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/14079/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/c9e3662dd468fe262fcb663ca1898cac0029f8d7","head":{"label":"squito:blacklist-SPARK-8425","ref":"blacklist-SPARK-8425","sha":"c9e3662dd468fe262fcb663ca1898cac0029f8d7","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"repo":{"id":29212749,"name":"spark","full_name":"squito/spark","owner":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/squito/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/squito/spark","forks_url":"https://api.github.com/repos/squito/spark/forks","keys_url":"https://api.github.com/repos/squito/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/squito/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/squito/spark/teams","hooks_url":"https://api.github.com/repos/squito/spark/hooks","issue_events_url":"https://api.github.com/repos/squito/spark/issues/events{/number}","events_url":"https://api.github.com/repos/squito/spark/events","assignees_url":"https://api.github.com/repos/squito/spark/assignees{/user}","branches_url":"https://api.github.com/repos/squito/spark/branches{/branch}","tags_url":"https://api.github.com/repos/squito/spark/tags","blobs_url":"https://api.github.com/repos/squito/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/squito/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/squito/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/squito/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/squito/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/squito/spark/languages","stargazers_url":"https://api.github.com/repos/squito/spark/stargazers","contributors_url":"https://api.github.com/repos/squito/spark/contributors","subscribers_url":"https://api.github.com/repos/squito/spark/subscribers","subscription_url":"https://api.github.com/repos/squito/spark/subscription","commits_url":"https://api.github.com/repos/squito/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/squito/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/squito/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/squito/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/squito/spark/contents/{+path}","compare_url":"https://api.github.com/repos/squito/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/squito/spark/merges","archive_url":"https://api.github.com/repos/squito/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/squito/spark/downloads","issues_url":"https://api.github.com/repos/squito/spark/issues{/number}","pulls_url":"https://api.github.com/repos/squito/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/squito/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/squito/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/squito/spark/labels{/name}","releases_url":"https://api.github.com/repos/squito/spark/releases{/id}","deployments_url":"https://api.github.com/repos/squito/spark/deployments","created_at":"2015-01-13T21:43:10Z","updated_at":"2015-05-06T03:00:35Z","pushed_at":"2016-07-26T16:21:02Z","git_url":"git://github.com/squito/spark.git","ssh_url":"git@github.com:squito/spark.git","clone_url":"https://github.com/squito/spark.git","svn_url":"https://github.com/squito/spark","homepage":null,"size":166774,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"f376c37268848dbb4b2fb57677e22ef2bf207b49","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-07-27T21:23:49Z","pushed_at":"2016-07-27T22:08:00Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":508404,"stargazers_count":9440,"watchers_count":9440,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":8756,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":440,"forks":8756,"open_issues":440,"watchers":9440,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/14079"},"html":{"href":"https://github.com/apache/spark/pull/14079"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/14079"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/14079/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/14079/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/14079/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/c9e3662dd468fe262fcb663ca1898cac0029f8d7"}}}}
{"actor":{"display_login":"kayousterhout"}}