{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21067","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21067/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21067/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21067/events","html_url":"https://github.com/apache/spark/pull/21067","id":314126939,"node_id":"MDExOlB1bGxSZXF1ZXN0MTgxNTAyODk1","number":21067,"title":"[SPARK-23980][K8S] Resilient Spark driver on Kubernetes","user":{"login":"baluchicken","id":14196407,"node_id":"MDQ6VXNlcjE0MTk2NDA3","avatar_url":"https://avatars0.githubusercontent.com/u/14196407?v=4","gravatar_id":"","url":"https://api.github.com/users/baluchicken","html_url":"https://github.com/baluchicken","followers_url":"https://api.github.com/users/baluchicken/followers","following_url":"https://api.github.com/users/baluchicken/following{/other_user}","gists_url":"https://api.github.com/users/baluchicken/gists{/gist_id}","starred_url":"https://api.github.com/users/baluchicken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/baluchicken/subscriptions","organizations_url":"https://api.github.com/users/baluchicken/orgs","repos_url":"https://api.github.com/users/baluchicken/repos","events_url":"https://api.github.com/users/baluchicken/events{/privacy}","received_events_url":"https://api.github.com/users/baluchicken/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":26,"created_at":"2018-04-13T14:21:53Z","updated_at":"2018-07-17T20:03:37Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21067","html_url":"https://github.com/apache/spark/pull/21067","diff_url":"https://github.com/apache/spark/pull/21067.diff","patch_url":"https://github.com/apache/spark/pull/21067.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThe current implementation of `Spark driver` on Kubernetes is not resilient to node failures as it’s implemented as a `Pod`. In case of a node failure Kubernetes terminates the pods that were running on that node. Kubernetes doesn't reschedule these pods to any of the other nodes of the cluster.\r\nIf the `driver` is implemented as Kubernetes Job than it will be rescheduled to other node.\r\nWhen the driver is terminated its executors (that may run on other nodes) are terminated by Kubernetes with some delay by Kubernetes Garbage collection.\r\nThis can lead to concurrency issues where the re-spawned `driver` was trying to create new executors with same name as the executors being in the middle of being cleaned up by Kubernetes garbage collection.\r\nTo solve this issue the executor name must be made unique for each `driver` instance.\r\nFor example: \r\n`networkwordcount-1519301591265-usmj-exec-1`\r\n\r\n## How was this patch tested?\r\n\r\nThis patch was tested manually.\r\nSubmitted a Spark application to a cluster with three node:\r\n\r\n```\r\nkubectl get jobs\r\nNAME                                    DESIRED   SUCCESSFUL   AGE\r\nnetworkwordcount-1519301591265-driver   1         0            3m\r\n```\r\n\r\n```\r\nkubectl get pods\r\nNAME                                          READY     STATUS    RESTARTS   AGE\r\nnetworkwordcount-1519301591265-driver-mszl2   1/1       Running   0          3m\r\nnetworkwordcount-1519301591265-usmj-exec-1    1/1       Running   0          1m\r\nnetworkwordcount-1519301591265-usmj-exec-2    1/1       Running   0          1m\r\n```\r\n\r\nSpark driver `networkwordcount-1519301591265-driver` is a Kubernetes Job, that manages the `networkwordcount-1519301591265-driver-mszl2` pod.\r\n\r\nShutted down the node where the driver pod is running\r\n\r\n```\r\nkubectl get pods\r\nNAME                                          READY     STATUS     RESTARTS   AGE\r\nnetworkwordcount-1519301591265-driver-dwvkf   1/1       Running   0          3m\r\nnetworkwordcount-1519301591265-rmes-exec-1    1/1       Running   0          1m\r\nnetworkwordcount-1519301591265-rmes-exec-2    1/1       Running   0          1m\r\n```\r\n\r\nThe spark driver kubernetes job rescheduled the driver pod as `networkwordcount-1519301591265-driver-dwvkf.\r\n`\r\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/405709862","html_url":"https://github.com/apache/spark/pull/21067#issuecomment-405709862","issue_url":"https://api.github.com/repos/apache/spark/issues/21067","id":405709862,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTcwOTg2Mg==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-07-17T20:03:37Z","updated_at":"2018-07-17T20:03:37Z","author_association":"NONE","body":"**[Test build #93201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93201/testReport)** for PR 21067 at commit [`c04179b`](https://github.com/apache/spark/commit/c04179b48056b14912a95016d5040777bdb1007c)."}}
{"actor":{"display_login":"SparkQA"}}