{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21748","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21748/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21748/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21748/events","html_url":"https://github.com/apache/spark/pull/21748","id":340347775,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAwNzgxNzUy","number":21748,"title":"[SPARK-23146][K8S] Support client mode.","user":{"login":"mccheah","id":8975385,"node_id":"MDQ6VXNlcjg5NzUzODU=","avatar_url":"https://avatars1.githubusercontent.com/u/8975385?v=4","gravatar_id":"","url":"https://api.github.com/users/mccheah","html_url":"https://github.com/mccheah","followers_url":"https://api.github.com/users/mccheah/followers","following_url":"https://api.github.com/users/mccheah/following{/other_user}","gists_url":"https://api.github.com/users/mccheah/gists{/gist_id}","starred_url":"https://api.github.com/users/mccheah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mccheah/subscriptions","organizations_url":"https://api.github.com/users/mccheah/orgs","repos_url":"https://api.github.com/users/mccheah/repos","events_url":"https://api.github.com/users/mccheah/events{/privacy}","received_events_url":"https://api.github.com/users/mccheah/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2018-07-11T18:02:55Z","updated_at":"2018-07-11T21:04:45Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21748","html_url":"https://github.com/apache/spark/pull/21748","diff_url":"https://github.com/apache/spark/pull/21748.diff","patch_url":"https://github.com/apache/spark/pull/21748.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSupport client mode for the Kubernetes scheduler.\r\n\r\nClient mode works more or less identically to cluster mode. However, in client mode, the Spark Context needs to be manually bootstrapped with certain properties which would have otherwise been set up by spark-submit in cluster mode. Specifically:\r\n\r\n- If the user doesn't provide a driver pod name, we don't add an owner reference. This is for usage when the driver is not running in a pod in the cluster. In such a case, the driver can only provide a best effort to clean up the executors when the driver exits, but cleaning up the resources is not guaranteed. The executor JVMs should exit if the driver JVM exits, but the pods will still remain in the cluster in a COMPLETED or FAILED state.\r\n- The user must provide a host (spark.driver.host) and port (spark.driver.port) that the executors can connect to. When using spark-submit in cluster mode, spark-submit generates the headless service automatically; in client mode, the user is responsible for setting up their own connectivity.\r\n\r\nWe also change the authentication configuration prefixes for client mode.\r\n\r\n## How was this patch tested?\r\n\r\nAdding an integration test to exercise client mode support.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/404308887","html_url":"https://github.com/apache/spark/pull/21748#issuecomment-404308887","issue_url":"https://api.github.com/repos/apache/spark/issues/21748","id":404308887,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDMwODg4Nw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-07-11T21:04:45Z","updated_at":"2018-07-11T21:04:45Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92892/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}