{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/23223","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/23223/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/23223/comments","events_url":"https://api.github.com/repos/apache/spark/issues/23223/events","html_url":"https://github.com/apache/spark/pull/23223","id":387609376,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM2MDU4OTI2","number":23223,"title":"[SPARK-26269][YARN]Yarnallocator should have same blacklist behaviour with yarn to maxmize use of cluster resource","user":{"login":"Ngone51","id":16397174,"node_id":"MDQ6VXNlcjE2Mzk3MTc0","avatar_url":"https://avatars1.githubusercontent.com/u/16397174?v=4","gravatar_id":"","url":"https://api.github.com/users/Ngone51","html_url":"https://github.com/Ngone51","followers_url":"https://api.github.com/users/Ngone51/followers","following_url":"https://api.github.com/users/Ngone51/following{/other_user}","gists_url":"https://api.github.com/users/Ngone51/gists{/gist_id}","starred_url":"https://api.github.com/users/Ngone51/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ngone51/subscriptions","organizations_url":"https://api.github.com/users/Ngone51/orgs","repos_url":"https://api.github.com/users/Ngone51/repos","events_url":"https://api.github.com/users/Ngone51/events{/privacy}","received_events_url":"https://api.github.com/users/Ngone51/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-12-05T06:17:50Z","updated_at":"2018-12-06T03:00:18Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/23223","html_url":"https://github.com/apache/spark/pull/23223","diff_url":"https://github.com/apache/spark/pull/23223.diff","patch_url":"https://github.com/apache/spark/pull/23223.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nAs I mentioned in jira [SPARK-26269](https://issues.apache.org/jira/browse/SPARK-26269), in order to maxmize the use of cluster resource,  this pr try to make `YarnAllocator` have the same blacklist behaviour with YARN.\r\n\r\n## How was this patch tested?\r\n\r\nAdded."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/444731361","html_url":"https://github.com/apache/spark/pull/23223#issuecomment-444731361","issue_url":"https://api.github.com/repos/apache/spark/issues/23223","id":444731361,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDczMTM2MQ==","user":{"login":"Ngone51","id":16397174,"node_id":"MDQ6VXNlcjE2Mzk3MTc0","avatar_url":"https://avatars1.githubusercontent.com/u/16397174?v=4","gravatar_id":"","url":"https://api.github.com/users/Ngone51","html_url":"https://github.com/Ngone51","followers_url":"https://api.github.com/users/Ngone51/followers","following_url":"https://api.github.com/users/Ngone51/following{/other_user}","gists_url":"https://api.github.com/users/Ngone51/gists{/gist_id}","starred_url":"https://api.github.com/users/Ngone51/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ngone51/subscriptions","organizations_url":"https://api.github.com/users/Ngone51/orgs","repos_url":"https://api.github.com/users/Ngone51/repos","events_url":"https://api.github.com/users/Ngone51/events{/privacy}","received_events_url":"https://api.github.com/users/Ngone51/received_events","type":"User","site_admin":false},"created_at":"2018-12-06T03:00:17Z","updated_at":"2018-12-06T03:00:17Z","author_association":"CONTRIBUTOR","body":"> I mean if node blacklisting in Spark would be perfectly aligned to YARN then it would be just redundant to have it in Spark in the first place. \r\n\r\nThis change seems result in *perfectly* aligned to YARN for node blacklisting in Spark, but my original thought is that some exit status (e.g. KILLED_BY_RESOURCEMANAGER ),  currently, should not lead to a node blacklisting. So, actually, *perfectly* aligned to YARN is not the real target of this change, and we can also make some custom strategy for Spark.\r\n\r\n> Take for example disk failure.\r\n\r\nFor spark  task level backlisting, is it should be delegated to **schedulerBlacklist** in YarnAllocatorBlacklistTracker ?\r\n\r\nAnd it seems ContainerExitStatus.DISKS_FAILED in YARN is not same with Spark tasks' disk failure."}}
{"actor":{"display_login":"Ngone51"}}