{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15249","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15249/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15249/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15249/events","html_url":"https://github.com/apache/spark/pull/15249","id":179380770,"number":15249,"title":"[SPARK-17675] [CORE] Expand Blacklist for TaskSets","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":32,"created_at":"2016-09-27T02:07:40Z","updated_at":"2016-10-06T19:17:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15249","html_url":"https://github.com/apache/spark/pull/15249","diff_url":"https://github.com/apache/spark/pull/15249.diff","patch_url":"https://github.com/apache/spark/pull/15249.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis is a step along the way to SPARK-8425.\r\n\r\nTo enable incremental review, the first step proposed here is to expand the blacklisting within tasksets. In particular, this will enable blacklisting for\r\n* (task, executor) pairs (this already exists via an undocumented config)\r\n* (task, node)\r\n* (taskset, executor)\r\n* (taskset, node)\r\n\r\nAdding (task, node) is critical to making spark fault-tolerant of one-bad disk in a cluster, without requiring careful tuning of \"spark.task.maxFailures\". The other additions are also important to avoid many misleading task failures and long scheduling delays when there is one bad node on a large cluster.\r\n\r\nNote that some of the code changes here aren't really required for just this -- they put pieces in place for SPARK-8425 even though they are not used yet (eg. the `BlacklistTracker` helper is a little out of place, `TaskSetBlacklist` holds onto a little more info than it needs to for just this change, and `ExecutorFailuresInTaskSet` is more complex than it needs to be).\r\n\r\n## How was this patch tested?\r\n\r\nAdded unit tests, run tests via jenkins."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/252061681","html_url":"https://github.com/apache/spark/pull/15249#issuecomment-252061681","issue_url":"https://api.github.com/repos/apache/spark/issues/15249","id":252061681,"user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"created_at":"2016-10-06T19:17:54Z","updated_at":"2016-10-06T19:17:54Z","body":"I agree with Kay's summary above, just one addition.  For (2) Temporary Resource Contention & the approach in this PR -- perhaps its obvious, but another consequence of this approach is that you lose resources for computing tasks, even if task-locality was never a consideration.  One of your executors is temporarily in trouble, so it fails a bunch of tasks, and then gets blacklisted from the entire taskset.  10 seconds later, its back to an OK state, but even if your taskset takes hours, you'd never take advantage of that other executor.\r\n\r\nI think the situation w/ (1) & this PR is fine.\r\n\r\nAlso I realized this was discussed in the design doc some under the [Flaky Apps Section](https://docs.google.com/document/d/1R2CVKctUZG9xwD67jkRdhBR4sCgccPR2dhTYSRXFEmg/edit#heading=h.3yb336nr3vy1) (not that it adds much more than what we have discussed here)."}}
{"actor":{"display_login":"squito"}}