{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21366","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21366/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21366/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21366/events","html_url":"https://github.com/apache/spark/pull/21366","id":324556557,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg5MTM4NDg4","number":21366,"title":"[SPARK-24248][K8S] Use the Kubernetes API to populate an event queue for scheduling","user":{"login":"mccheah","id":8975385,"node_id":"MDQ6VXNlcjg5NzUzODU=","avatar_url":"https://avatars1.githubusercontent.com/u/8975385?v=4","gravatar_id":"","url":"https://api.github.com/users/mccheah","html_url":"https://github.com/mccheah","followers_url":"https://api.github.com/users/mccheah/followers","following_url":"https://api.github.com/users/mccheah/following{/other_user}","gists_url":"https://api.github.com/users/mccheah/gists{/gist_id}","starred_url":"https://api.github.com/users/mccheah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mccheah/subscriptions","organizations_url":"https://api.github.com/users/mccheah/orgs","repos_url":"https://api.github.com/users/mccheah/repos","events_url":"https://api.github.com/users/mccheah/events{/privacy}","received_events_url":"https://api.github.com/users/mccheah/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":92,"created_at":"2018-05-18T21:24:15Z","updated_at":"2018-06-07T18:38:38Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21366","html_url":"https://github.com/apache/spark/pull/21366","diff_url":"https://github.com/apache/spark/pull/21366.diff","patch_url":"https://github.com/apache/spark/pull/21366.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nPreviously, the scheduler backend was maintaining state in many places, not only for reading state but also writing to it. For example, state had to be managed in both the watch and in the executor allocator runnable. Furthermore, one had to keep track of multiple hash tables.\r\n\r\nWe can do better here by:\r\n\r\n1. Consolidating the places where we manage state. Here, we take inspiration from traditional Kubernetes controllers. These controllers tend to implement an event queue which is populated by two sources: a watch connection, and a periodic poller. Controllers typically use both mechanisms for redundancy; the watch connection may drop, so the periodic polling serves as a backup. Both sources write pod updates to a single event queue and then a processor periodically processes the current state of pods as reported by the two sources.\r\n\r\n2. Storing less specialized in-memory state in general. Previously we were creating hash tables to represent the state of executors. Instead, it's easier to represent state solely by the event queue, which has predictable read/write patterns and is more or less just a local up-to-date cache of the cluster's status.\r\n\r\n## How was this patch tested?\r\n\r\nIntegration tests should test there's no regressions end to end. Unit tests to be updated, in particular focusing on different orderings of events, particularly accounting for when events come in unexpected ordering.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/395522740","html_url":"https://github.com/apache/spark/pull/21366#issuecomment-395522740","issue_url":"https://api.github.com/repos/apache/spark/issues/21366","id":395522740,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTUyMjc0MA==","user":{"login":"foxish","id":906471,"node_id":"MDQ6VXNlcjkwNjQ3MQ==","avatar_url":"https://avatars1.githubusercontent.com/u/906471?v=4","gravatar_id":"","url":"https://api.github.com/users/foxish","html_url":"https://github.com/foxish","followers_url":"https://api.github.com/users/foxish/followers","following_url":"https://api.github.com/users/foxish/following{/other_user}","gists_url":"https://api.github.com/users/foxish/gists{/gist_id}","starred_url":"https://api.github.com/users/foxish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/foxish/subscriptions","organizations_url":"https://api.github.com/users/foxish/orgs","repos_url":"https://api.github.com/users/foxish/repos","events_url":"https://api.github.com/users/foxish/events{/privacy}","received_events_url":"https://api.github.com/users/foxish/received_events","type":"User","site_admin":false},"created_at":"2018-06-07T18:38:38Z","updated_at":"2018-06-07T18:38:38Z","author_association":"CONTRIBUTOR","body":"If we did miss the watch as you said, when the next poll comes along, that should capture the fact that a pod that we thought existed doesn't exist anymore in the API. So, that would be the level triggered approach - where we'll see the expected state diverge from the current state and take action."}}
{"actor":{"display_login":"foxish"}}