{"action":"closed","number":37,"pull_request":{"url":"https://api.github.com/repos/lessor/lessor/pulls/37","id":193797066,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkzNzk3MDY2","html_url":"https://github.com/lessor/lessor/pull/37","diff_url":"https://github.com/lessor/lessor/pull/37.diff","patch_url":"https://github.com/lessor/lessor/pull/37.patch","issue_url":"https://api.github.com/repos/lessor/lessor/issues/37","number":37,"state":"closed","locked":false,"title":"Hard Multi-Tenancy Design and Implementation Ramblings","user":{"login":"marpaia","id":927168,"node_id":"MDQ6VXNlcjkyNzE2OA==","avatar_url":"https://avatars2.githubusercontent.com/u/927168?v=4","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"body":"> This PR contains a design doc which contains a summary of the @jessfraz's design for hard multi-tenancy thus far, some ramblings about implementation, and some challenges/consequences. Many thanks to @davidopp as well, for his questions and clarifications on the doc, which are incorporated here where relevant.\r\n>\r\n> I've copied the content of the markdown here and I'll try to keep it up to date as I edit the doc, but please comment on the PR itself if you have feedback or clarification.\r\n>\r\n---\r\n\r\n# Hard Multi-Tenancy Design and Implementation Ramblings\r\n\r\n## Authors\r\n\r\n- Mike Arpaia ([@marpaia](https://github.com/marpaia))\r\n\r\n## Context\r\n\r\nIn [Hard Multi-Tenancy in Kubernetes](https://docs.google.com/document/d/1mNL5oCIqtVwXI9piTPMuGArdZH8CA2UFaxHtM5Myp6M/edit#), Jessie Frazelle wrote about a design for \"hard multi-tenancy\" and how it may work in Kubernetes. In this document, I’d like to agree on an objective design and set some initial development objectives.\r\n\r\nDavid Oppenheimer said the following rather succinctly in a comment on Jessie’s doc:\r\n\r\n> I think there are two ways to look at this architecture. One is that it's a way to provide strong isolation between tenants within a cluster, by nesting per-tenant clusters within a host cluster. (That's the way it's described here--start with a cluster and add isolation boundaries.)\r\n>\r\n> The other way to look at it is that you start with independent per-tenant clusters (the typical way people do strong isolation between tenants today) and ask how can you make that more resource-efficient. When you start from that direction I think you end up in a very similar place with respect to the control plane components, i.e. hosting the control planes for all of the clusters on a shared cluster (see for example https://kubernetes.io/blog/2018/05/17/gardener/). Of course that still leaves you with separate physical nodes, so it doesn't get you all the way to the third approach described here.\r\n\r\nJessie's doc outlines 3 levels of organization that have varying degrees of isolation and resource efficiency. For the sake of keeping things interesting, I'm going to assume that we've all decided that we should go with the third design (with some modifications based on comments).\r\n\r\nThe key modification is based on a comment by David, where he added that API server and system services should be able to take advantage of whatever isolation is being provided to other nodes.\r\n\r\nThe resultant distribution of workloads thus looks something like this:\r\n\r\n```\r\n+------------+ +------------+  +------------+ +-------------+  +------------+ +------------+\r\n| Tenant Foo | | Tenant Bar |  | Tenant Foo | | Tenant Bar  |  | Tenant Foo | | Tenant Bar |\r\n| API Server | |    Pod     |  |    Pod     | | API Server  |  |    Pod     | |    Pod     |\r\n+------------+ +------------+  +------------+ +-------------+  +------------+ +------------+\r\n+------------+ +------------+  +------------+ +-------------+  +------------+ +------------+\r\n| Tenant Bar | | Tenant Bar |  | Tenant Bar | | Tenant Foo  |  | Tenant Bar | | Tenant Bar |\r\n|    Pod     | |    Pod     |  |    Pod     | |    Pod      |  |    Pod     | |    Pod     |\r\n+------------+ +------------+  +------------+ +-------------+  +------------+ +------------+\r\n+------------+ +------------+  +------------+ +-------------+  +------------+ +------------+\r\n| Tenant Foo | |   Ring0    |  | Tenant Bar | |    Ring0    |  | Tenant Foo | |    Ring0   |\r\n|    Pod     | |  Kubelet   |  |    Pod     | |   Kubelet   |  |    Pod     | |   Kubelet  |\r\n+------------+ +------------+  +------------+ +-------------+  +------------+ +------------+\r\n+---------------------------+  +----------------------------+  +---------------------------+\r\n|          Node 1           |  |          Node 2            |  |          Node 3           |\r\n+---------------------------+  +----------------------------+  +---------------------------+\r\n+------------------------------------------------------------------------------------------+\r\n|                                     Ring0 Kubernetes                                     |\r\n+------------------------------------------------------------------------------------------+\r\n```\r\n\r\n## Potential Implementation\r\n\r\nIf we consider implementing a controller which manages an API Server (including all required components) per \"tenant\" as described in the Context section above using the Kubernetes Operator pattern, then as with any \"operator\", the role of the CRD is to define objective state. So if we think that we should do the whole Kubernetes in Kubernetes thing, the role of the controller is then to:\r\n\r\n- Automate the management of a Kubernetes cluster per tenant, running on Kubernetes\r\n  - API Server, Etcd, etc\r\n- Effectively configure authentication and authorization to the appropriate API server per-user, per-tenant\r\n\r\nThe main runloop of the Lessor controller can be found in [`pkg/controller/synchronize.go`](../../pkg/controller/synchronize.go) in the `resolveTenantState` function. Whenever a tenant resource is updated, this function processes the tenant resource and endeavors to converge desired and actual state.\r\n\r\n```go\r\n// resolveTenantState compares the actual state with the desired, and attempts to\r\n// converge the two. It then updates the Status block of the tenant resource\r\n// with the current status of the resource.\r\nfunc (c *Controller) resolveTenantState(key string) error {\r\n\ttenant, ok, err := c.tenantForCacheKey(key)\r\n\t// check err and ok\r\n\r\n\t// given the tenant variable which contains the full Kubernetes resource,\r\n\t// do stuff like:\r\n\t// - ensure all API components are running properly\r\n\t// - ensure authnz is configured correctly\r\n```\r\n\r\n## Consequences\r\n\r\nThe biggest consequence of this decision as far as I can tell is the Kubelet. This design calls for a single \"Ring0 Kubelet\" on each Ring0 Node to be communicating with multiple API servers (n+1, where n is the number of tenants). This is challenging because, as enumerated by David on Jessie's doc, there are a whole mess of problems with how various parts of Kubernetes work today that would complicate a design like this. Before this design can be considered further, more extensive research needs to be put into the feasibility of some of the changes that would need to various actions that are currently very API server centric.","created_at":"2018-06-09T21:28:29Z","updated_at":"2018-06-29T20:35:25Z","closed_at":"2018-06-29T20:35:24Z","merged_at":"2018-06-29T20:35:24Z","merge_commit_sha":"9dedb71a7f979783152e5f7c1e981429c74cf6d0","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/lessor/lessor/pulls/37/commits","review_comments_url":"https://api.github.com/repos/lessor/lessor/pulls/37/comments","review_comment_url":"https://api.github.com/repos/lessor/lessor/pulls/comments{/number}","comments_url":"https://api.github.com/repos/lessor/lessor/issues/37/comments","statuses_url":"https://api.github.com/repos/lessor/lessor/statuses/f75a475a2acd93554d431508ebf28ad009c145a2","head":{"label":"marpaia:design-doc","ref":"design-doc","sha":"f75a475a2acd93554d431508ebf28ad009c145a2","user":{"login":"marpaia","id":927168,"node_id":"MDQ6VXNlcjkyNzE2OA==","avatar_url":"https://avatars2.githubusercontent.com/u/927168?v=4","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"repo":{"id":136648607,"node_id":"MDEwOlJlcG9zaXRvcnkxMzY2NDg2MDc=","name":"lessor","full_name":"marpaia/lessor","owner":{"login":"marpaia","id":927168,"node_id":"MDQ6VXNlcjkyNzE2OA==","avatar_url":"https://avatars2.githubusercontent.com/u/927168?v=4","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/marpaia/lessor","description":"Deploy, manage, and secure applications on Kubernetes","fork":true,"url":"https://api.github.com/repos/marpaia/lessor","forks_url":"https://api.github.com/repos/marpaia/lessor/forks","keys_url":"https://api.github.com/repos/marpaia/lessor/keys{/key_id}","collaborators_url":"https://api.github.com/repos/marpaia/lessor/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/marpaia/lessor/teams","hooks_url":"https://api.github.com/repos/marpaia/lessor/hooks","issue_events_url":"https://api.github.com/repos/marpaia/lessor/issues/events{/number}","events_url":"https://api.github.com/repos/marpaia/lessor/events","assignees_url":"https://api.github.com/repos/marpaia/lessor/assignees{/user}","branches_url":"https://api.github.com/repos/marpaia/lessor/branches{/branch}","tags_url":"https://api.github.com/repos/marpaia/lessor/tags","blobs_url":"https://api.github.com/repos/marpaia/lessor/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/marpaia/lessor/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/marpaia/lessor/git/refs{/sha}","trees_url":"https://api.github.com/repos/marpaia/lessor/git/trees{/sha}","statuses_url":"https://api.github.com/repos/marpaia/lessor/statuses/{sha}","languages_url":"https://api.github.com/repos/marpaia/lessor/languages","stargazers_url":"https://api.github.com/repos/marpaia/lessor/stargazers","contributors_url":"https://api.github.com/repos/marpaia/lessor/contributors","subscribers_url":"https://api.github.com/repos/marpaia/lessor/subscribers","subscription_url":"https://api.github.com/repos/marpaia/lessor/subscription","commits_url":"https://api.github.com/repos/marpaia/lessor/commits{/sha}","git_commits_url":"https://api.github.com/repos/marpaia/lessor/git/commits{/sha}","comments_url":"https://api.github.com/repos/marpaia/lessor/comments{/number}","issue_comment_url":"https://api.github.com/repos/marpaia/lessor/issues/comments{/number}","contents_url":"https://api.github.com/repos/marpaia/lessor/contents/{+path}","compare_url":"https://api.github.com/repos/marpaia/lessor/compare/{base}...{head}","merges_url":"https://api.github.com/repos/marpaia/lessor/merges","archive_url":"https://api.github.com/repos/marpaia/lessor/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/marpaia/lessor/downloads","issues_url":"https://api.github.com/repos/marpaia/lessor/issues{/number}","pulls_url":"https://api.github.com/repos/marpaia/lessor/pulls{/number}","milestones_url":"https://api.github.com/repos/marpaia/lessor/milestones{/number}","notifications_url":"https://api.github.com/repos/marpaia/lessor/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/marpaia/lessor/labels{/name}","releases_url":"https://api.github.com/repos/marpaia/lessor/releases{/id}","deployments_url":"https://api.github.com/repos/marpaia/lessor/deployments","created_at":"2018-06-08T17:36:37Z","updated_at":"2018-06-12T06:34:38Z","pushed_at":"2018-06-29T20:28:10Z","git_url":"git://github.com/marpaia/lessor.git","ssh_url":"git@github.com:marpaia/lessor.git","clone_url":"https://github.com/marpaia/lessor.git","svn_url":"https://github.com/marpaia/lessor","homepage":"","size":2979,"stargazers_count":1,"watchers_count":1,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"lessor:master","ref":"master","sha":"6a7d6fd73353ab2a39cd2ef2c22940ca8a113668","user":{"login":"lessor","id":37767103,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM3NzY3MTAz","avatar_url":"https://avatars1.githubusercontent.com/u/37767103?v=4","gravatar_id":"","url":"https://api.github.com/users/lessor","html_url":"https://github.com/lessor","followers_url":"https://api.github.com/users/lessor/followers","following_url":"https://api.github.com/users/lessor/following{/other_user}","gists_url":"https://api.github.com/users/lessor/gists{/gist_id}","starred_url":"https://api.github.com/users/lessor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lessor/subscriptions","organizations_url":"https://api.github.com/users/lessor/orgs","repos_url":"https://api.github.com/users/lessor/repos","events_url":"https://api.github.com/users/lessor/events{/privacy}","received_events_url":"https://api.github.com/users/lessor/received_events","type":"Organization","site_admin":false},"repo":{"id":126726627,"node_id":"MDEwOlJlcG9zaXRvcnkxMjY3MjY2Mjc=","name":"lessor","full_name":"lessor/lessor","owner":{"login":"lessor","id":37767103,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM3NzY3MTAz","avatar_url":"https://avatars1.githubusercontent.com/u/37767103?v=4","gravatar_id":"","url":"https://api.github.com/users/lessor","html_url":"https://github.com/lessor","followers_url":"https://api.github.com/users/lessor/followers","following_url":"https://api.github.com/users/lessor/following{/other_user}","gists_url":"https://api.github.com/users/lessor/gists{/gist_id}","starred_url":"https://api.github.com/users/lessor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lessor/subscriptions","organizations_url":"https://api.github.com/users/lessor/orgs","repos_url":"https://api.github.com/users/lessor/repos","events_url":"https://api.github.com/users/lessor/events{/privacy}","received_events_url":"https://api.github.com/users/lessor/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/lessor/lessor","description":"A Kubernetes Operator for managing multi-tenant workloads","fork":false,"url":"https://api.github.com/repos/lessor/lessor","forks_url":"https://api.github.com/repos/lessor/lessor/forks","keys_url":"https://api.github.com/repos/lessor/lessor/keys{/key_id}","collaborators_url":"https://api.github.com/repos/lessor/lessor/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/lessor/lessor/teams","hooks_url":"https://api.github.com/repos/lessor/lessor/hooks","issue_events_url":"https://api.github.com/repos/lessor/lessor/issues/events{/number}","events_url":"https://api.github.com/repos/lessor/lessor/events","assignees_url":"https://api.github.com/repos/lessor/lessor/assignees{/user}","branches_url":"https://api.github.com/repos/lessor/lessor/branches{/branch}","tags_url":"https://api.github.com/repos/lessor/lessor/tags","blobs_url":"https://api.github.com/repos/lessor/lessor/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/lessor/lessor/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/lessor/lessor/git/refs{/sha}","trees_url":"https://api.github.com/repos/lessor/lessor/git/trees{/sha}","statuses_url":"https://api.github.com/repos/lessor/lessor/statuses/{sha}","languages_url":"https://api.github.com/repos/lessor/lessor/languages","stargazers_url":"https://api.github.com/repos/lessor/lessor/stargazers","contributors_url":"https://api.github.com/repos/lessor/lessor/contributors","subscribers_url":"https://api.github.com/repos/lessor/lessor/subscribers","subscription_url":"https://api.github.com/repos/lessor/lessor/subscription","commits_url":"https://api.github.com/repos/lessor/lessor/commits{/sha}","git_commits_url":"https://api.github.com/repos/lessor/lessor/git/commits{/sha}","comments_url":"https://api.github.com/repos/lessor/lessor/comments{/number}","issue_comment_url":"https://api.github.com/repos/lessor/lessor/issues/comments{/number}","contents_url":"https://api.github.com/repos/lessor/lessor/contents/{+path}","compare_url":"https://api.github.com/repos/lessor/lessor/compare/{base}...{head}","merges_url":"https://api.github.com/repos/lessor/lessor/merges","archive_url":"https://api.github.com/repos/lessor/lessor/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/lessor/lessor/downloads","issues_url":"https://api.github.com/repos/lessor/lessor/issues{/number}","pulls_url":"https://api.github.com/repos/lessor/lessor/pulls{/number}","milestones_url":"https://api.github.com/repos/lessor/lessor/milestones{/number}","notifications_url":"https://api.github.com/repos/lessor/lessor/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/lessor/lessor/labels{/name}","releases_url":"https://api.github.com/repos/lessor/lessor/releases{/id}","deployments_url":"https://api.github.com/repos/lessor/lessor/deployments","created_at":"2018-03-25T18:20:08Z","updated_at":"2018-06-28T22:54:14Z","pushed_at":"2018-06-29T20:35:24Z","git_url":"git://github.com/lessor/lessor.git","ssh_url":"git@github.com:lessor/lessor.git","clone_url":"https://github.com/lessor/lessor.git","svn_url":"https://github.com/lessor/lessor","homepage":"","size":2975,"stargazers_count":22,"watchers_count":22,"language":"Go","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":6,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":6,"open_issues":0,"watchers":22,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/lessor/lessor/pulls/37"},"html":{"href":"https://github.com/lessor/lessor/pull/37"},"issue":{"href":"https://api.github.com/repos/lessor/lessor/issues/37"},"comments":{"href":"https://api.github.com/repos/lessor/lessor/issues/37/comments"},"review_comments":{"href":"https://api.github.com/repos/lessor/lessor/pulls/37/comments"},"review_comment":{"href":"https://api.github.com/repos/lessor/lessor/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/lessor/lessor/pulls/37/commits"},"statuses":{"href":"https://api.github.com/repos/lessor/lessor/statuses/f75a475a2acd93554d431508ebf28ad009c145a2"}},"author_association":"MEMBER","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"marpaia","id":927168,"node_id":"MDQ6VXNlcjkyNzE2OA==","avatar_url":"https://avatars2.githubusercontent.com/u/927168?v=4","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"comments":1,"review_comments":0,"maintainer_can_modify":false,"commits":1,"additions":142,"deletions":0,"changed_files":2}}
{"actor":{"display_login":"marpaia"}}