{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/195513995","pull_request_review_id":128899224,"id":195513995,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5NTUxMzk5NQ==","diff_hunk":"@@ -56,17 +58,44 @@ private[spark] class KubernetesClusterManager extends ExternalClusterManager wit\n       Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)),\n       Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)))\n \n-    val allocatorExecutor = ThreadUtils\n-      .newDaemonSingleThreadScheduledExecutor(\"kubernetes-pod-allocator\")\n     val requestExecutorsService = ThreadUtils.newDaemonCachedThreadPool(\n       \"kubernetes-executor-requests\")\n+\n+    val bufferSnapshotsExecutor = ThreadUtils\n+      .newDaemonSingleThreadScheduledExecutor(\"kubernetes-executor-snapshots-buffer\")\n+    val snapshotsStore = new ExecutorPodsSnapshotsStoreImpl(bufferSnapshotsExecutor)\n+    val removedExecutorsCache = CacheBuilder.newBuilder()\n+      .expireAfterWrite(3, TimeUnit.MINUTES)","path":"resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala","position":32,"original_position":32,"commit_id":"8b0a211f4c5046065b0e2abcac0ee71119ff6bd6","original_commit_id":"8b0a211f4c5046065b0e2abcac0ee71119ff6bd6","user":{"login":"mccheah","id":8975385,"node_id":"MDQ6VXNlcjg5NzUzODU=","avatar_url":"https://avatars1.githubusercontent.com/u/8975385?v=4","gravatar_id":"","url":"https://api.github.com/users/mccheah","html_url":"https://github.com/mccheah","followers_url":"https://api.github.com/users/mccheah/followers","following_url":"https://api.github.com/users/mccheah/following{/other_user}","gists_url":"https://api.github.com/users/mccheah/gists{/gist_id}","starred_url":"https://api.github.com/users/mccheah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mccheah/subscriptions","organizations_url":"https://api.github.com/users/mccheah/orgs","repos_url":"https://api.github.com/users/mccheah/repos","events_url":"https://api.github.com/users/mccheah/events{/privacy}","received_events_url":"https://api.github.com/users/mccheah/received_events","type":"User","site_admin":false},"body":"The cache is only for a best effort attempt to not remove the same executor from the scheduler backend multiple times, but at the end of the day even if we do accidentally remove multiple times the only noticeable result is noisy logs. The scheduler backend properly handles multiple attempts to remove but we'd prefer it if we didn't have to.","created_at":"2018-06-14T17:49:22Z","updated_at":"2018-06-14T17:49:22Z","html_url":"https://github.com/apache/spark/pull/21366#discussion_r195513995","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/21366","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/195513995"},"html":{"href":"https://github.com/apache/spark/pull/21366#discussion_r195513995"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/21366"}},"in_reply_to_id":195379060},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21366","id":189138488,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg5MTM4NDg4","html_url":"https://github.com/apache/spark/pull/21366","diff_url":"https://github.com/apache/spark/pull/21366.diff","patch_url":"https://github.com/apache/spark/pull/21366.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/21366","number":21366,"state":"open","locked":false,"title":"[SPARK-24248][K8S] Use level triggering and state reconciliation in scheduling and lifecycle","user":{"login":"mccheah","id":8975385,"node_id":"MDQ6VXNlcjg5NzUzODU=","avatar_url":"https://avatars1.githubusercontent.com/u/8975385?v=4","gravatar_id":"","url":"https://api.github.com/users/mccheah","html_url":"https://github.com/mccheah","followers_url":"https://api.github.com/users/mccheah/followers","following_url":"https://api.github.com/users/mccheah/following{/other_user}","gists_url":"https://api.github.com/users/mccheah/gists{/gist_id}","starred_url":"https://api.github.com/users/mccheah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mccheah/subscriptions","organizations_url":"https://api.github.com/users/mccheah/orgs","repos_url":"https://api.github.com/users/mccheah/repos","events_url":"https://api.github.com/users/mccheah/events{/privacy}","received_events_url":"https://api.github.com/users/mccheah/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nPreviously, the scheduler backend was maintaining state in many places, not only for reading state but also writing to it. For example, state had to be managed in both the watch and in the executor allocator runnable. Furthermore, one had to keep track of multiple hash tables.\r\n\r\nWe can do better here by:\r\n\r\n1. Consolidating the places where we manage state. Here, we take inspiration from traditional Kubernetes controllers. These controllers tend to follow a level-triggered mechanism. This means that the controller will continuously monitor the API server via watches and polling, and on periodic passes, the controller will reconcile the current state of the cluster with the desired state. We implement this by introducing the concept of a pod snapshot, which is a given state of the executors in the Kubernetes cluster. We operate periodically on snapshots. To prevent overloading the API server with polling requests to get the state of the cluster (particularly for executor allocation where we want to be checking frequently to get executors to launch without unbearably bad latency), we use watches to populate snapshots by applying observed events to a previous snapshot to get a new snapshot. Whenever we do poll the cluster, the polled state replaces any existing snapshot - this ensures eventual consistency and mirroring of the cluster, as is desired in a level triggered architecture.\r\n\r\n2. Storing less specialized in-memory state in general. Previously we were creating hash tables to represent the state of executors. Instead, it's easier to represent state solely by the snapshots.\r\n\r\n## How was this patch tested?\r\n\r\nIntegration tests should test there's no regressions end to end. Unit tests to be updated, in particular focusing on different orderings of events, particularly accounting for when events come in unexpected ordering.\r\n","created_at":"2018-05-18T21:24:15Z","updated_at":"2018-06-14T17:49:22Z","closed_at":null,"merged_at":null,"merge_commit_sha":"21a96a3b9666854d4ae9b914a328a98055987bf3","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/21366/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/21366/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/21366/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/8b0a211f4c5046065b0e2abcac0ee71119ff6bd6","head":{"label":"palantir:event-queue-driven-scheduling","ref":"event-queue-driven-scheduling","sha":"8b0a211f4c5046065b0e2abcac0ee71119ff6bd6","user":{"login":"palantir","id":303157,"node_id":"MDEyOk9yZ2FuaXphdGlvbjMwMzE1Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/303157?v=4","gravatar_id":"","url":"https://api.github.com/users/palantir","html_url":"https://github.com/palantir","followers_url":"https://api.github.com/users/palantir/followers","following_url":"https://api.github.com/users/palantir/following{/other_user}","gists_url":"https://api.github.com/users/palantir/gists{/gist_id}","starred_url":"https://api.github.com/users/palantir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palantir/subscriptions","organizations_url":"https://api.github.com/users/palantir/orgs","repos_url":"https://api.github.com/users/palantir/repos","events_url":"https://api.github.com/users/palantir/events{/privacy}","received_events_url":"https://api.github.com/users/palantir/received_events","type":"Organization","site_admin":false},"repo":{"id":30919203,"node_id":"MDEwOlJlcG9zaXRvcnkzMDkxOTIwMw==","name":"spark","full_name":"palantir/spark","owner":{"login":"palantir","id":303157,"node_id":"MDEyOk9yZ2FuaXphdGlvbjMwMzE1Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/303157?v=4","gravatar_id":"","url":"https://api.github.com/users/palantir","html_url":"https://github.com/palantir","followers_url":"https://api.github.com/users/palantir/followers","following_url":"https://api.github.com/users/palantir/following{/other_user}","gists_url":"https://api.github.com/users/palantir/gists{/gist_id}","starred_url":"https://api.github.com/users/palantir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palantir/subscriptions","organizations_url":"https://api.github.com/users/palantir/orgs","repos_url":"https://api.github.com/users/palantir/repos","events_url":"https://api.github.com/users/palantir/events{/privacy}","received_events_url":"https://api.github.com/users/palantir/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/palantir/spark","description":"Palantir Distribution of Apache Spark","fork":true,"url":"https://api.github.com/repos/palantir/spark","forks_url":"https://api.github.com/repos/palantir/spark/forks","keys_url":"https://api.github.com/repos/palantir/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/palantir/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/palantir/spark/teams","hooks_url":"https://api.github.com/repos/palantir/spark/hooks","issue_events_url":"https://api.github.com/repos/palantir/spark/issues/events{/number}","events_url":"https://api.github.com/repos/palantir/spark/events","assignees_url":"https://api.github.com/repos/palantir/spark/assignees{/user}","branches_url":"https://api.github.com/repos/palantir/spark/branches{/branch}","tags_url":"https://api.github.com/repos/palantir/spark/tags","blobs_url":"https://api.github.com/repos/palantir/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/palantir/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/palantir/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/palantir/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/palantir/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/palantir/spark/languages","stargazers_url":"https://api.github.com/repos/palantir/spark/stargazers","contributors_url":"https://api.github.com/repos/palantir/spark/contributors","subscribers_url":"https://api.github.com/repos/palantir/spark/subscribers","subscription_url":"https://api.github.com/repos/palantir/spark/subscription","commits_url":"https://api.github.com/repos/palantir/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/palantir/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/palantir/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/palantir/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/palantir/spark/contents/{+path}","compare_url":"https://api.github.com/repos/palantir/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/palantir/spark/merges","archive_url":"https://api.github.com/repos/palantir/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/palantir/spark/downloads","issues_url":"https://api.github.com/repos/palantir/spark/issues{/number}","pulls_url":"https://api.github.com/repos/palantir/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/palantir/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/palantir/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/palantir/spark/labels{/name}","releases_url":"https://api.github.com/repos/palantir/spark/releases{/id}","deployments_url":"https://api.github.com/repos/palantir/spark/deployments","created_at":"2015-02-17T13:57:35Z","updated_at":"2018-06-07T17:08:47Z","pushed_at":"2018-06-14T10:42:18Z","git_url":"git://github.com/palantir/spark.git","ssh_url":"git@github.com:palantir/spark.git","clone_url":"https://github.com/palantir/spark.git","svn_url":"https://github.com/palantir/spark","homepage":"","size":273968,"stargazers_count":24,"watchers_count":24,"language":"Scala","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":20,"mirror_url":null,"archived":false,"open_issues_count":17,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":20,"open_issues":17,"watchers":24,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"f433ef786770e48e3594ad158ce9908f98ef0d9a","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"node_id":"MDEwOlJlcG9zaXRvcnkxNzE2NTY1OA==","name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2018-06-14T17:01:34Z","pushed_at":"2018-06-14T17:41:48Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":271788,"stargazers_count":17689,"watchers_count":17689,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":16121,"mirror_url":null,"archived":false,"open_issues_count":567,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":16121,"open_issues":567,"watchers":17689,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/21366"},"html":{"href":"https://github.com/apache/spark/pull/21366"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/21366"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/21366/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/21366/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/21366/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/8b0a211f4c5046065b0e2abcac0ee71119ff6bd6"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"mccheah"}}