{"action":"created","comment":{"url":"https://api.github.com/repos/openshift/origin/pulls/comments/52661962","id":52661962,"diff_hunk":"@@ -0,0 +1,114 @@\n+# Emptydir Volume Quotas\n+\n+## Abstract\n+\n+This proposal describes a new volume plugin which implements storage quota for\n+emptyDir volume types. The volume plugin is designed as a drop-in replacement\n+for the emptydir plugin.\n+\n+## Constraints and Assumptions\n+\n+1. This proposal describes a quota implementation which expresses limits at the\n+namespace level. Pod-scoped quota is possible but not addressed here.\n+\n+2. XFS must be available on nodes for which quota must be enforced.\n+\n+3. No scheduler integration is assumed. Overcommit must be avoided through other\n+capacity restraints. (For example, by enforcing a fixed quota size across all\n+namespaces and limiting the number of pods per node through CPU, memory,\n+cadvisor capacity information, etc.)\n+\n+## Design\n+\n+Each OpenShift project is assigned a cluster-unique FSGroup ID which is\n+propagated to the security context of all pods and made accessible to volume\n+plugins.\n+\n+Building on these features, the new volume plugin can configure XFS quota on the\n+node during volume setup to restrict disk usage per namespace per node. With\n+this plugin, disk usage of all pods within a namespace per node would count\n+towards the namespace's quota on that node.\n+\n+For example, given:\n+\n+1. A cluster configuration which dictates a fixed 100 MB emptyDir quota for\n+namespaces\n+2. Two nodes *node-1* and *node-2* each with 50 GB allocated for emptyDir\n+storage\n+3. The following cluster state:\n+\n+| Pod        | Node   |\n+|------------|--------|\n+| ns1/pod-a  | node-1 |\n+| ns1/pod-b  | node-1 |\n+| ns2/pod-a  | node-1 |\n+| ns1/pod-c  | node-2 |\n+\n+Any data written an emptyDir volume by *ns1/pod-a* or *ns1/pod-b* share a 100 MB\n+quota on *node-1*, while *ns1/pod-c* emptyDir data counts towards a 100 MB quota\n+on *node-2*. In total, *ns1* can consume up to 200 MB of emptyDir storage on\n+this cluster.\n+\n+It's important to note that there is no explicit overcommit prevention through\n+scheduling. If the scenario is modified such that *node-1* has only 100 MB of\n+storage, *ns2/pod-a* may still be scheduled to *node-1* and depending on the\n+underlying storage configuration, the disk backing all emptyDirs for the node\n+could be filled.\n+\n+### XFS Implementation\n+\n+To support the quota implementation, the volumes backing emptyDir storage\n+on each node must be mounted with XFS quota enabled:\n+\n+```shell\n+mount -o gquota /dev/sdb2 /var/openshift/volumes/emptydir\n+```\n+\n+The volume plugin implementation will use `xfs_quota` to constrain disk for a\n+given FSGroupID. For example, the plugin may issue a command such as:\n+\n+```shell\n+xfs_quota -x -c 'limit -g bsoft=120m bhard=128m $FSGROUPID' /var/openshift/volumes/emptydir/$FSGROUPID\n+```\n+\n+Repeated quota application on the same directory has no known issues.\n+\n+Pods with the same FSGroupID will have a common root directly, e.g.\n+`/var/openshift/volumes/emptydir/$FSGROUPID/$POD_UID`.\n+\n+#### Caveats\n+\n+* The XFS quota tools don't provide any means to delete a specific group's quota\n+configuration. The rules are stored on disk, and so the maximum number of quota\n+entries is limited by inode/disk space on the node. Even with a high rate of\n+quota-enabled pod churn it seems unlikely these limits will be reached during\n+the normal course of operation.","path":"docs/proposals/emptydir-quota.md","position":85,"original_position":85,"commit_id":"aeb578c82d0f9278b1db14920636a393b3fd972c","original_commit_id":"aeb578c82d0f9278b1db14920636a393b3fd972c","user":{"login":"ironcladlou","id":298299,"avatar_url":"https://avatars.githubusercontent.com/u/298299?v=3","gravatar_id":"","url":"https://api.github.com/users/ironcladlou","html_url":"https://github.com/ironcladlou","followers_url":"https://api.github.com/users/ironcladlou/followers","following_url":"https://api.github.com/users/ironcladlou/following{/other_user}","gists_url":"https://api.github.com/users/ironcladlou/gists{/gist_id}","starred_url":"https://api.github.com/users/ironcladlou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironcladlou/subscriptions","organizations_url":"https://api.github.com/users/ironcladlou/orgs","repos_url":"https://api.github.com/users/ironcladlou/repos","events_url":"https://api.github.com/users/ironcladlou/events{/privacy}","received_events_url":"https://api.github.com/users/ironcladlou/received_events","type":"User","site_admin":false},"body":"@dgoodwin did the up-front investigation into how quota entries are stored and determined that it's on disk, and my current impression is that those entries are consumed by stateless utilities (and not held in-memory by the kernel, etc), but we should definitely dig deeper. I'd like to be able to definitively say there is no memory leak and no impact beyond the stateless userland tooling.","created_at":"2016-02-11T20:29:21Z","updated_at":"2016-02-11T20:29:21Z","html_url":"https://github.com/openshift/origin/pull/7206#discussion_r52661962","pull_request_url":"https://api.github.com/repos/openshift/origin/pulls/7206","_links":{"self":{"href":"https://api.github.com/repos/openshift/origin/pulls/comments/52661962"},"html":{"href":"https://github.com/openshift/origin/pull/7206#discussion_r52661962"},"pull_request":{"href":"https://api.github.com/repos/openshift/origin/pulls/7206"}}},"pull_request":{"url":"https://api.github.com/repos/openshift/origin/pulls/7206","id":58960373,"html_url":"https://github.com/openshift/origin/pull/7206","diff_url":"https://github.com/openshift/origin/pull/7206.diff","patch_url":"https://github.com/openshift/origin/pull/7206.patch","issue_url":"https://api.github.com/repos/openshift/origin/issues/7206","number":7206,"state":"open","locked":false,"title":"WIP: EmptyDir volume quota proposal","user":{"login":"ironcladlou","id":298299,"avatar_url":"https://avatars.githubusercontent.com/u/298299?v=3","gravatar_id":"","url":"https://api.github.com/users/ironcladlou","html_url":"https://github.com/ironcladlou","followers_url":"https://api.github.com/users/ironcladlou/followers","following_url":"https://api.github.com/users/ironcladlou/following{/other_user}","gists_url":"https://api.github.com/users/ironcladlou/gists{/gist_id}","starred_url":"https://api.github.com/users/ironcladlou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironcladlou/subscriptions","organizations_url":"https://api.github.com/users/ironcladlou/orgs","repos_url":"https://api.github.com/users/ironcladlou/repos","events_url":"https://api.github.com/users/ironcladlou/events{/privacy}","received_events_url":"https://api.github.com/users/ironcladlou/received_events","type":"User","site_admin":false},"body":"## :construction: WIP :construction: \r\n\r\nIntroduce a proposal for applying disk quota to emptyDir volumes backed by XFS.\r\n\r\nThis is a WIP proposal to try and capture the design in a format more conducive to collaboration. It's largely a distillation of R&D and prototyping work already done by @dgoodwin.","created_at":"2016-02-10T21:57:44Z","updated_at":"2016-02-11T20:29:22Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6a7a8827280a72ba35545b150b2d18edb63812d6","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/openshift/origin/pulls/7206/commits","review_comments_url":"https://api.github.com/repos/openshift/origin/pulls/7206/comments","review_comment_url":"https://api.github.com/repos/openshift/origin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/openshift/origin/issues/7206/comments","statuses_url":"https://api.github.com/repos/openshift/origin/statuses/aeb578c82d0f9278b1db14920636a393b3fd972c","head":{"label":"ironcladlou:emptydir-quota-proposal","ref":"emptydir-quota-proposal","sha":"aeb578c82d0f9278b1db14920636a393b3fd972c","user":{"login":"ironcladlou","id":298299,"avatar_url":"https://avatars.githubusercontent.com/u/298299?v=3","gravatar_id":"","url":"https://api.github.com/users/ironcladlou","html_url":"https://github.com/ironcladlou","followers_url":"https://api.github.com/users/ironcladlou/followers","following_url":"https://api.github.com/users/ironcladlou/following{/other_user}","gists_url":"https://api.github.com/users/ironcladlou/gists{/gist_id}","starred_url":"https://api.github.com/users/ironcladlou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironcladlou/subscriptions","organizations_url":"https://api.github.com/users/ironcladlou/orgs","repos_url":"https://api.github.com/users/ironcladlou/repos","events_url":"https://api.github.com/users/ironcladlou/events{/privacy}","received_events_url":"https://api.github.com/users/ironcladlou/received_events","type":"User","site_admin":false},"repo":{"id":23721352,"name":"origin","full_name":"ironcladlou/origin","owner":{"login":"ironcladlou","id":298299,"avatar_url":"https://avatars.githubusercontent.com/u/298299?v=3","gravatar_id":"","url":"https://api.github.com/users/ironcladlou","html_url":"https://github.com/ironcladlou","followers_url":"https://api.github.com/users/ironcladlou/followers","following_url":"https://api.github.com/users/ironcladlou/following{/other_user}","gists_url":"https://api.github.com/users/ironcladlou/gists{/gist_id}","starred_url":"https://api.github.com/users/ironcladlou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironcladlou/subscriptions","organizations_url":"https://api.github.com/users/ironcladlou/orgs","repos_url":"https://api.github.com/users/ironcladlou/repos","events_url":"https://api.github.com/users/ironcladlou/events{/privacy}","received_events_url":"https://api.github.com/users/ironcladlou/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ironcladlou/origin","description":"OpenShift 3 - build, deploy, and manage your applications with Docker and Kubernetes","fork":true,"url":"https://api.github.com/repos/ironcladlou/origin","forks_url":"https://api.github.com/repos/ironcladlou/origin/forks","keys_url":"https://api.github.com/repos/ironcladlou/origin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ironcladlou/origin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ironcladlou/origin/teams","hooks_url":"https://api.github.com/repos/ironcladlou/origin/hooks","issue_events_url":"https://api.github.com/repos/ironcladlou/origin/issues/events{/number}","events_url":"https://api.github.com/repos/ironcladlou/origin/events","assignees_url":"https://api.github.com/repos/ironcladlou/origin/assignees{/user}","branches_url":"https://api.github.com/repos/ironcladlou/origin/branches{/branch}","tags_url":"https://api.github.com/repos/ironcladlou/origin/tags","blobs_url":"https://api.github.com/repos/ironcladlou/origin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ironcladlou/origin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ironcladlou/origin/git/refs{/sha}","trees_url":"https://api.github.com/repos/ironcladlou/origin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ironcladlou/origin/statuses/{sha}","languages_url":"https://api.github.com/repos/ironcladlou/origin/languages","stargazers_url":"https://api.github.com/repos/ironcladlou/origin/stargazers","contributors_url":"https://api.github.com/repos/ironcladlou/origin/contributors","subscribers_url":"https://api.github.com/repos/ironcladlou/origin/subscribers","subscription_url":"https://api.github.com/repos/ironcladlou/origin/subscription","commits_url":"https://api.github.com/repos/ironcladlou/origin/commits{/sha}","git_commits_url":"https://api.github.com/repos/ironcladlou/origin/git/commits{/sha}","comments_url":"https://api.github.com/repos/ironcladlou/origin/comments{/number}","issue_comment_url":"https://api.github.com/repos/ironcladlou/origin/issues/comments{/number}","contents_url":"https://api.github.com/repos/ironcladlou/origin/contents/{+path}","compare_url":"https://api.github.com/repos/ironcladlou/origin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ironcladlou/origin/merges","archive_url":"https://api.github.com/repos/ironcladlou/origin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ironcladlou/origin/downloads","issues_url":"https://api.github.com/repos/ironcladlou/origin/issues{/number}","pulls_url":"https://api.github.com/repos/ironcladlou/origin/pulls{/number}","milestones_url":"https://api.github.com/repos/ironcladlou/origin/milestones{/number}","notifications_url":"https://api.github.com/repos/ironcladlou/origin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ironcladlou/origin/labels{/name}","releases_url":"https://api.github.com/repos/ironcladlou/origin/releases{/id}","deployments_url":"https://api.github.com/repos/ironcladlou/origin/deployments","created_at":"2014-09-05T23:22:09Z","updated_at":"2016-01-12T22:03:53Z","pushed_at":"2016-02-11T16:14:25Z","git_url":"git://github.com/ironcladlou/origin.git","ssh_url":"git@github.com:ironcladlou/origin.git","clone_url":"https://github.com/ironcladlou/origin.git","svn_url":"https://github.com/ironcladlou/origin","homepage":"https://openshift.github.io","size":205106,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"openshift:master","ref":"master","sha":"8301f6b263b039d9031d4384fcf25a716f3c6497","user":{"login":"openshift","id":792337,"avatar_url":"https://avatars.githubusercontent.com/u/792337?v=3","gravatar_id":"","url":"https://api.github.com/users/openshift","html_url":"https://github.com/openshift","followers_url":"https://api.github.com/users/openshift/followers","following_url":"https://api.github.com/users/openshift/following{/other_user}","gists_url":"https://api.github.com/users/openshift/gists{/gist_id}","starred_url":"https://api.github.com/users/openshift/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openshift/subscriptions","organizations_url":"https://api.github.com/users/openshift/orgs","repos_url":"https://api.github.com/users/openshift/repos","events_url":"https://api.github.com/users/openshift/events{/privacy}","received_events_url":"https://api.github.com/users/openshift/received_events","type":"Organization","site_admin":false},"repo":{"id":22442668,"name":"origin","full_name":"openshift/origin","owner":{"login":"openshift","id":792337,"avatar_url":"https://avatars.githubusercontent.com/u/792337?v=3","gravatar_id":"","url":"https://api.github.com/users/openshift","html_url":"https://github.com/openshift","followers_url":"https://api.github.com/users/openshift/followers","following_url":"https://api.github.com/users/openshift/following{/other_user}","gists_url":"https://api.github.com/users/openshift/gists{/gist_id}","starred_url":"https://api.github.com/users/openshift/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openshift/subscriptions","organizations_url":"https://api.github.com/users/openshift/orgs","repos_url":"https://api.github.com/users/openshift/repos","events_url":"https://api.github.com/users/openshift/events{/privacy}","received_events_url":"https://api.github.com/users/openshift/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/openshift/origin","description":"OpenShift 3 - build, deploy, and manage your applications with Docker and Kubernetes","fork":false,"url":"https://api.github.com/repos/openshift/origin","forks_url":"https://api.github.com/repos/openshift/origin/forks","keys_url":"https://api.github.com/repos/openshift/origin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/openshift/origin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/openshift/origin/teams","hooks_url":"https://api.github.com/repos/openshift/origin/hooks","issue_events_url":"https://api.github.com/repos/openshift/origin/issues/events{/number}","events_url":"https://api.github.com/repos/openshift/origin/events","assignees_url":"https://api.github.com/repos/openshift/origin/assignees{/user}","branches_url":"https://api.github.com/repos/openshift/origin/branches{/branch}","tags_url":"https://api.github.com/repos/openshift/origin/tags","blobs_url":"https://api.github.com/repos/openshift/origin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/openshift/origin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/openshift/origin/git/refs{/sha}","trees_url":"https://api.github.com/repos/openshift/origin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/openshift/origin/statuses/{sha}","languages_url":"https://api.github.com/repos/openshift/origin/languages","stargazers_url":"https://api.github.com/repos/openshift/origin/stargazers","contributors_url":"https://api.github.com/repos/openshift/origin/contributors","subscribers_url":"https://api.github.com/repos/openshift/origin/subscribers","subscription_url":"https://api.github.com/repos/openshift/origin/subscription","commits_url":"https://api.github.com/repos/openshift/origin/commits{/sha}","git_commits_url":"https://api.github.com/repos/openshift/origin/git/commits{/sha}","comments_url":"https://api.github.com/repos/openshift/origin/comments{/number}","issue_comment_url":"https://api.github.com/repos/openshift/origin/issues/comments{/number}","contents_url":"https://api.github.com/repos/openshift/origin/contents/{+path}","compare_url":"https://api.github.com/repos/openshift/origin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/openshift/origin/merges","archive_url":"https://api.github.com/repos/openshift/origin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/openshift/origin/downloads","issues_url":"https://api.github.com/repos/openshift/origin/issues{/number}","pulls_url":"https://api.github.com/repos/openshift/origin/pulls{/number}","milestones_url":"https://api.github.com/repos/openshift/origin/milestones{/number}","notifications_url":"https://api.github.com/repos/openshift/origin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/openshift/origin/labels{/name}","releases_url":"https://api.github.com/repos/openshift/origin/releases{/id}","deployments_url":"https://api.github.com/repos/openshift/origin/deployments","created_at":"2014-07-30T20:04:05Z","updated_at":"2016-02-11T18:10:27Z","pushed_at":"2016-02-11T20:18:02Z","git_url":"git://github.com/openshift/origin.git","ssh_url":"git@github.com:openshift/origin.git","clone_url":"https://github.com/openshift/origin.git","svn_url":"https://github.com/openshift/origin","homepage":"http://www.openshift.org","size":266443,"stargazers_count":1025,"watchers_count":1025,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":627,"mirror_url":null,"open_issues_count":830,"forks":627,"open_issues":830,"watchers":1025,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/openshift/origin/pulls/7206"},"html":{"href":"https://github.com/openshift/origin/pull/7206"},"issue":{"href":"https://api.github.com/repos/openshift/origin/issues/7206"},"comments":{"href":"https://api.github.com/repos/openshift/origin/issues/7206/comments"},"review_comments":{"href":"https://api.github.com/repos/openshift/origin/pulls/7206/comments"},"review_comment":{"href":"https://api.github.com/repos/openshift/origin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/openshift/origin/pulls/7206/commits"},"statuses":{"href":"https://api.github.com/repos/openshift/origin/statuses/aeb578c82d0f9278b1db14920636a393b3fd972c"}}}}