{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/81423415","pull_request_review_id":2050961,"id":81423415,"diff_hunk":"@@ -809,32 +816,65 @@ class TaskSetManagerSuite extends SparkFunSuite with LocalSparkContext with Logg\n   test(\"Kill other task attempts when one attempt belonging to the same task succeeds\") {\n     sc = new SparkContext(\"local\", \"test\")\n     sched = new FakeTaskScheduler(sc, (\"exec1\", \"host1\"), (\"exec2\", \"host2\"))\n-    val taskSet = FakeTask.createTaskSet(4)\n+    val taskSet = FakeTask.createTaskSet(5)\n     // Set the speculation multiplier to be 0 so speculative tasks are launched immediately\n     sc.conf.set(\"spark.speculation.multiplier\", \"0.0\")\n+    sc.conf.set(\"spark.speculation.quantile\", \"0.6\")\n     val clock = new ManualClock()\n     val manager = new TaskSetManager(sched, taskSet, MAX_TASK_FAILURES, clock)\n     val accumUpdatesByTask: Array[Seq[AccumulatorV2[_, _]]] = taskSet.tasks.map { task =>\n       task.metrics.internalAccums\n     }\n     // Offer resources for 4 tasks to start\n+    val tasks = new ArrayBuffer[TaskDescription]()\n     for ((k, v) <- List(\n         \"exec1\" -> \"host1\",\n         \"exec1\" -> \"host1\",\n+        \"exec1\" -> \"host1\",\n         \"exec2\" -> \"host2\",\n         \"exec2\" -> \"host2\")) {\n       val taskOption = manager.resourceOffer(k, v, NO_PREF)\n       assert(taskOption.isDefined)\n       val task = taskOption.get\n       assert(task.executorId === k)\n+      tasks += task\n     }\n-    assert(sched.startedTasks.toSet === Set(0, 1, 2, 3))\n-    // Complete the 3 tasks and leave 1 task in running\n+    assert(sched.startedTasks.toSet === (0 until 5).toSet)\n+    // Complete 3 tasks and leave 2 task in running\n     for (id <- Set(0, 1, 2)) {\n       manager.handleSuccessfulTask(id, createTaskResult(id, accumUpdatesByTask(id)))\n       assert(sched.endedTasks(id) === Success)\n     }\n \n+    def runningTaskForIndex(index: Int): TaskDescription = {\n+      val t = tasks.find { task => task.index == index && !sched.endedTasks.contains(task.taskId) }","path":"core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala","position":104,"original_position":104,"commit_id":"9c9d8165cc0d220d511ea1855d3f53d97277dce1","original_commit_id":"9c9d8165cc0d220d511ea1855d3f53d97277dce1","user":{"login":"kayousterhout","id":1108612,"avatar_url":"https://avatars.githubusercontent.com/u/1108612?v=3","gravatar_id":"","url":"https://api.github.com/users/kayousterhout","html_url":"https://github.com/kayousterhout","followers_url":"https://api.github.com/users/kayousterhout/followers","following_url":"https://api.github.com/users/kayousterhout/following{/other_user}","gists_url":"https://api.github.com/users/kayousterhout/gists{/gist_id}","starred_url":"https://api.github.com/users/kayousterhout/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kayousterhout/subscriptions","organizations_url":"https://api.github.com/users/kayousterhout/orgs","repos_url":"https://api.github.com/users/kayousterhout/repos","events_url":"https://api.github.com/users/kayousterhout/events{/privacy}","received_events_url":"https://api.github.com/users/kayousterhout/received_events","type":"User","site_admin":false},"body":"here you can do tasks.find {... }.getOrElse { throw new RuntimeException(s\".... }","created_at":"2016-09-30T21:46:18Z","updated_at":"2016-09-30T21:48:39Z","html_url":"https://github.com/apache/spark/pull/15249#discussion_r81423415","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/15249","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/81423415"},"html":{"href":"https://github.com/apache/spark/pull/15249#discussion_r81423415"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/15249"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15249","id":86850892,"html_url":"https://github.com/apache/spark/pull/15249","diff_url":"https://github.com/apache/spark/pull/15249.diff","patch_url":"https://github.com/apache/spark/pull/15249.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/15249","number":15249,"state":"open","locked":false,"title":"[SPARK-17675] [CORE] Expand Blacklist for TaskSets","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis is a step along the way to SPARK-8425.\r\n\r\nTo enable incremental review, the first step proposed here is to expand the blacklisting within tasksets. In particular, this will enable blacklisting for\r\n* (task, executor) pairs (this already exists via an undocumented config)\r\n* (task, node)\r\n* (taskset, executor)\r\n* (taskset, node)\r\n\r\nAdding (task, node) is critical to making spark fault-tolerant of one-bad disk in a cluster, without requiring careful tuning of \"spark.task.maxFailures\". The other additions are also important to avoid many misleading task failures and long scheduling delays when there is one bad node on a large cluster.\r\n\r\nNote that some of the code changes here aren't really required for just this -- they put pieces in place for SPARK-8425 even though they are not used yet (eg. the `BlacklistTracker` helper is a little out of place, and `TaskSetBlacklist` holds onto a little more info than it needs to for just this change).\r\n\r\n## How was this patch tested?\r\n\r\nAdded unit tests, run tests via jenkins.","created_at":"2016-09-27T02:07:40Z","updated_at":"2016-09-30T21:48:40Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e0f548f88961477f9f6621140c158cfd4ebca6aa","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/15249/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/15249/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/15249/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/9c9d8165cc0d220d511ea1855d3f53d97277dce1","head":{"label":"squito:taskset_blacklist_only","ref":"taskset_blacklist_only","sha":"9c9d8165cc0d220d511ea1855d3f53d97277dce1","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"repo":{"id":29212749,"name":"spark","full_name":"squito/spark","owner":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/squito/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/squito/spark","forks_url":"https://api.github.com/repos/squito/spark/forks","keys_url":"https://api.github.com/repos/squito/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/squito/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/squito/spark/teams","hooks_url":"https://api.github.com/repos/squito/spark/hooks","issue_events_url":"https://api.github.com/repos/squito/spark/issues/events{/number}","events_url":"https://api.github.com/repos/squito/spark/events","assignees_url":"https://api.github.com/repos/squito/spark/assignees{/user}","branches_url":"https://api.github.com/repos/squito/spark/branches{/branch}","tags_url":"https://api.github.com/repos/squito/spark/tags","blobs_url":"https://api.github.com/repos/squito/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/squito/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/squito/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/squito/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/squito/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/squito/spark/languages","stargazers_url":"https://api.github.com/repos/squito/spark/stargazers","contributors_url":"https://api.github.com/repos/squito/spark/contributors","subscribers_url":"https://api.github.com/repos/squito/spark/subscribers","subscription_url":"https://api.github.com/repos/squito/spark/subscription","commits_url":"https://api.github.com/repos/squito/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/squito/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/squito/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/squito/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/squito/spark/contents/{+path}","compare_url":"https://api.github.com/repos/squito/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/squito/spark/merges","archive_url":"https://api.github.com/repos/squito/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/squito/spark/downloads","issues_url":"https://api.github.com/repos/squito/spark/issues{/number}","pulls_url":"https://api.github.com/repos/squito/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/squito/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/squito/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/squito/spark/labels{/name}","releases_url":"https://api.github.com/repos/squito/spark/releases{/id}","deployments_url":"https://api.github.com/repos/squito/spark/deployments","created_at":"2015-01-13T21:43:10Z","updated_at":"2015-05-06T03:00:35Z","pushed_at":"2016-09-30T15:05:15Z","git_url":"git://github.com/squito/spark.git","ssh_url":"git@github.com:squito/spark.git","clone_url":"https://github.com/squito/spark.git","svn_url":"https://github.com/squito/spark","homepage":null,"size":170010,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"1fad5596885aab8b32d2307c0edecbae50d5bd7a","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-09-30T17:23:45Z","pushed_at":"2016-09-30T21:23:18Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":524767,"stargazers_count":10214,"watchers_count":10214,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":9501,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":486,"forks":9501,"open_issues":486,"watchers":10214,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/15249"},"html":{"href":"https://github.com/apache/spark/pull/15249"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/15249"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/15249/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/15249/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/15249/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/9c9d8165cc0d220d511ea1855d3f53d97277dce1"}}}}
{"actor":{"display_login":"kayousterhout"}}