{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19267","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19267/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19267/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19267/events","html_url":"https://github.com/apache/spark/pull/19267","id":258547505,"node_id":"MDExOlB1bGxSZXF1ZXN0MTQxNjUxNTA4","number":19267,"title":"[WIP][SPARK-20628][CORE] Blacklist nodes when they transition to DECOMMISSIONING state in YARN","user":{"login":"juanrh","id":895341,"node_id":"MDQ6VXNlcjg5NTM0MQ==","avatar_url":"https://avatars3.githubusercontent.com/u/895341?v=4","gravatar_id":"","url":"https://api.github.com/users/juanrh","html_url":"https://github.com/juanrh","followers_url":"https://api.github.com/users/juanrh/followers","following_url":"https://api.github.com/users/juanrh/following{/other_user}","gists_url":"https://api.github.com/users/juanrh/gists{/gist_id}","starred_url":"https://api.github.com/users/juanrh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juanrh/subscriptions","organizations_url":"https://api.github.com/users/juanrh/orgs","repos_url":"https://api.github.com/users/juanrh/repos","events_url":"https://api.github.com/users/juanrh/events{/privacy}","received_events_url":"https://api.github.com/users/juanrh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2017-09-18T16:47:04Z","updated_at":"2018-10-22T06:40:25Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19267","html_url":"https://github.com/apache/spark/pull/19267","diff_url":"https://github.com/apache/spark/pull/19267.diff","patch_url":"https://github.com/apache/spark/pull/19267.patch"},"body":"## What changes were proposed in this pull request? \r\nDynamic cluster configurations where cluster nodes are added and removed frequently are common in public cloud environments, so this has become a [problem for Spark users](https://www.trulia.com/blog/tech/aws-emr-ad-hoc-spark-development-environme... ). To cope with this we propose implementing a mechanism in the line of YARN’s support for [graceful node decommission](https://issues.apache.org/jira/browse/YARN-914 ) or [Mesos maintenance primitives](https://mesos.apache.org/documentation/latest/maintenance/ ). These changes allow cluster nodes to be transitioned to a ‘decommissioning’ state, at which point no more tasks will be scheduled on the executors running on those nodes. After a configurable drain time, nodes in the ‘decommissioning’ state will transition to a ‘decommissioned’ state, where shuffle blocks are not available anymore. Shuffle blocks stored on nodes in the ‘decommissioning’ state are available to other executors. By preventing more tasks from running on nodes in the ‘decommissioning’ state we avoid creating more shuffle blocks on those nodes, as those blocks won’t be available when nodes eventually transition to the ‘decommissioned’ state. \r\n\r\nWe have implemented a first version of this proposal for YARN, using Spark’s [blacklisting mechanism for task scheduling](https://issues.apache.org/jira/browse/SPARK-8425 ) ─available at the node level since Spark 2.2.0─ to ensure tasks are not scheduled on nodes in the ‘decommissioning’ state. With this solution it is the cluster manager, not the Spark application, that tracks the status of the node, and handles the transition from ‘decommissioning’ to ‘decommissioned’. The Spark driver simply reacts to the node state transitions. \r\n\r\n## How was this patch tested? \r\nAll functionality has been tested with unit tests, with an integration test based on the BaseYarnClusterSuite, and with manual testing on a cluster "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/431751864","html_url":"https://github.com/apache/spark/pull/19267#issuecomment-431751864","issue_url":"https://api.github.com/repos/apache/spark/issues/19267","id":431751864,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTc1MTg2NA==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T06:40:25Z","updated_at":"2018-10-22T06:40:25Z","author_association":"NONE","body":"Can one of the admins verify this patch?"}}
{"actor":{"display_login":"AmplabJenkins"}}