{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13603","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13603/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13603/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13603/events","html_url":"https://github.com/apache/spark/pull/13603","id":159677849,"number":13603,"title":"[SPARK-15865][CORE] Blacklist should not result in job hanging with less than 4 executors","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-10T16:55:29Z","updated_at":"2016-06-10T20:01:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13603","html_url":"https://github.com/apache/spark/pull/13603","diff_url":"https://github.com/apache/spark/pull/13603.diff","patch_url":"https://github.com/apache/spark/pull/13603.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nBefore this change, when you turn on blacklisting with `spark.scheduler.executorTaskBlacklistTime`, but you have fewer than `spark.task.maxFailures` executors, you can end with a job \"hung\" after some task failures.\r\n\r\nWhenever a taskset is unable to schedule anything on resourceOfferSingleTaskSet, we check whether the last pending task can be scheduled on *any* known executor.  If not, the taskset (and any corresponding jobs) are failed.\r\n* Worst case, this is O(numExecutors).  But unless many executors are bad, this should be small\r\n* This does not fail as fast as possible -- when a task becomes unschedulable, we keep scheduling other tasks.  This is to avoid an O(numPendingTasks) operation\r\n* Also, it is conceivable this fails too quickly.  You may be 1 millisecond away from unblacklisting a place for a task to run, or acquiring a new executor.\r\n\r\nAlso, I scratched an itch and tightened up visibility throughout `TaskSetManager`.  Can undo that if it clutters this change.\r\n\r\n## How was this patch tested?\r\n\r\nAdded unit test which failed before the change, ran new test 5k times manually, ran all scheduler tests manually, and the full suite via jenkins."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/225281379","html_url":"https://github.com/apache/spark/pull/13603#issuecomment-225281379","issue_url":"https://api.github.com/repos/apache/spark/issues/13603","id":225281379,"user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"created_at":"2016-06-10T20:01:41Z","updated_at":"2016-06-10T20:01:41Z","body":"q: was blacklist merged in 2.0?\r\n"}}
{"actor":{"display_login":"rxin"}}