{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/55458921","id":55458921,"diff_hunk":"@@ -0,0 +1,496 @@\n+<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->\n+\n+<!-- BEGIN STRIP_FOR_RELEASE -->\n+\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+\n+<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>\n+\n+If you are using a released version of Kubernetes, you should\n+refer to the docs that go with that version.\n+\n+Documentation for other releases can be found at\n+[releases.k8s.io](http://releases.k8s.io).\n+</strong>\n+--\n+\n+<!-- END STRIP_FOR_RELEASE -->\n+\n+<!-- END MUNGE: UNVERSIONED_WARNING -->\n+\n+# Controlled Rescheduling in Kubernetes\n+\n+## Overview\n+\n+Although the Kubernetes scheduler(s) try to make good placement decisions for pods,\n+conditions in the cluster change over time (e.g. jobs finish and new pods arrive, nodes\n+are removed due to failures or planned maintenance or auto-scaling down, nodes appear due\n+to recovery after a failure or re-joining after maintenance or auto-scaling up or adding\n+new hardware to a bare-metal cluster), and schedulers are not omniscient (e.g. there are\n+some interactions between pods, or between pods and nodes, that they cannot predict). As\n+a result, the initial node selected for a pod may turn out to be a bad match, from the\n+perspective of the pod and/or the cluster as a whole, at some point after the pod has\n+started running.\n+\n+Today (Kubernetes version 1.2) once a pod is scheduled to a node, it never moves unless\n+it terminates on its own, is deleted by the user, or experiences some unplanned event\n+(e.g. the node where it is running dies). Thus in a cluster with long-running pods, the\n+assignment of pods to nodes degrades over time, no matter how good an initial scheduling\n+decision the scheduler makes. This observation motivates \"controlled rescheduling,\" a\n+mechanism by which Kubernetes will \"move\" already-running pods over time to improve their\n+placement. Controlled rescheduling is the subject of this proposal.\n+\n+Note that the term \"move\" is not technically accurate -- the mechanism used is that\n+Kubernetes will terminate a pod that is managed by a controller, and the controller will\n+create a replacement pod that is then scheduled by the pod's scheduler. The terminated\n+pod and replacement pod are completely separate pods, and no pod migration is\n+implied. However, describing the process as \"moving\" the pod is approximately accurate\n+and easier to understand, so we will use this terminology in the document.\n+\n+We use the term \"rescheduling\" to describe any action the system takes to move an\n+already-running pod. The decision may be made and executed by any component; we wil\n+introduce the concept of a \"rescheduler\" component later, but it is not the only\n+component that can do rescheduling.\n+\n+This proposal primarily focuses on the architecture and features/mechanisms used to\n+achieve rescheduling, and only briefly discuss example policies. We expect that community\n+experimentation will lead to a significantly better understanding of the range, potential,\n+and limitations of rescheduling policies.\n+\n+## Example use cases\n+\n+Example use cases for rescheduling are\n+* moving a running pod onto a node that better satisfies its scheduling criteria\n+  * moving a pod onto an under-utilized node\n+  * moving a pod onto a node that meets more of the pod's affinity/anti-affinity preferences\n+* moving a running pod off of a node in anticipation of a known or speculated future event\n+  * draining a node in preparation for maintenance, decomissioning, auto-scale-down, etc.\n+  * \"preempting\" a running pod to make room for a pending pod to schedule\n+  * proactively/speculatively make room for large and/or exclusive pods to facilitate\n+    fast scheduling in the future (often called \"defragmentation\")\n+  * (note that these last two cases are the only use cases where the first-order intent\n+    is to move a pod specifically for the benefit of another pod)\n+* moving a running pod off of a node from which it is receiving poor service\n+  * anomalous crashlooping or other mysterious incompatiblity between the pod and the node\n+  * repeated out-of-resource killing (see #18724)\n+  * repeated attempts by the scheduler to schedule the pod onto some node, but it is\n+    rejected by Kubelet admission control due to incomplete scheduler knowledge\n+  * poor performance due to interference from other containers on the node (CPU hogs,\n+    cache thrashers, etc.) (note that in this case there is a choice of moving the victim\n+    or the aggressor)\n+\n+## Some axes of the design space\n+\n+Among the key design decisions are\n+* how does a pod specify its tolerance for these system-generated disruptions, and how\n+  does the system enforce such disruption limits\n+* for each use case, where is the decision made about when and which pods to reschedule\n+  (controllers, schedulers, an entirely new component e.g. \"rescheduler\", etc.)\n+* rescheduler design issues: how much does a rescheduler need to know about pods'\n+  schedulers' policies, how does the rescheduler specify its rescheduling\n+  requests/decisions (e.g. just as an eviction, an eviction with a hint about where to\n+  reschedule, or as an eviction paired with a specific binding), how does the system\n+  implement these requests, does the rescheduler take into account the second-order\n+  effects of decisions (e.g. whether an evicted pod will reschedule, will cause\n+  a preemption when it reschedules, etc.), does the rescheduler execute multi-step plans\n+  (e.g. evict two pods at the same time with the intent of moving one into the space\n+  vacated by the other, or even more complex plans)\n+\n+Additional musings on the rescheduling design space can be found [here](rescheduler.md).\n+\n+## Design proposal\n+\n+The key mechanisms and components of the proposed design are priority, preemption,\n+disruption budgets, the `/evict` subresource, and the rescheduler.\n+\n+### Priority\n+\n+#### Motivation\n+\n+[Quality of Service levels](resource-qos.md) allow Kubernetes to safely overcommit nodes.\n+Just as it is useful to overcommit nodes to increase node-level utilization, it is useful\n+to overcommit clusters to increase cluster-level utilization.  Scheduling priority (which\n+we abbreviate as *priority*, in combination with disruption budgets (described in the\n+next section), allows Kubernetes to safely overcommit clusters much as QoS levels allow\n+it to safely overcommit nodes.\n+\n+Today, cluster sharing among users, workload types, etc. is regulated via the\n+[quota](../admin/resourcequota/README.md) mechanism. When allocating quota, a cluster\n+administrator has two choices: (1) the sum of the quotas is less than or equal to the\n+capacity of the cluster, or (2) the sum of the quotas is greater than the capacity of the\n+cluster (that is, the cluster is overcommitted).  (1) is likely to lead to cluster\n+under-utilization, while (2) is unsafe in the sense that someone's pods may go pending\n+indefinitely even though they are still within their quota. Priority makes cluster\n+overcommitment (i.e. case (2)) safe by allowing users and/or administrators to identify\n+which pods should be allowed to run, and which should go pending, when demand for cluster\n+resources exceeds supply to due to cluster overcommitment.\n+\n+Priority is also useful in some special-cause scenarios, such as ensuring that system\n+DaemonSets can always schedule and reschedule onto every node where they want to run\n+(assuming they are given the highest priority), e.g. see #21767.\n+\n+#### Specifying priorities\n+\n+We propose to add a required `Priority` field to `PodSpec`. Its value type is string, and\n+the cluster administrator defines a total ordering on these strings (for example\n+`Critical`, `Normal`, `Preemptible`). We choose string instead of integer so that it is\n+easy for an administrator to add new priority levels in between existing levels, to\n+encourage thinking about priority in terms of user intent and avoid magic numbers, and to\n+make the internal implementation more flexible.\n+\n+When a scheduler is scheduling a new pod P and cannot find any node that meets all of P's\n+scheduling predicates, it is allowed to evict (\"preempt\") one or more pods that are at\n+the same or lower priority than P (subject to disruption budgets, see next section) from\n+a node in order to make room for P, i.e. in order to make the scheduling predicates\n+satisfied for P on that node.  (Note that when we add cluster-level resources (#19080),\n+it might be necessary to preempt from multiple nodes, but that scenario is outside the\n+scope of this document.)  The preempted pod(s) may or may not be able to reschedule. The\n+net effect of this process is that when demand for cluster resources exceeds supply, the\n+higher-priority pods will be able to run while the lower-priority pods will be forced to\n+wait. The detailed mechanics of preemption are described in a later section.\n+\n+In addition to taking disruption budget into account, for equal-priority preemptions the\n+scheduler will try to enforce fairness (across victim controllers, services, etc.)\n+\n+Priorities could be specified directly by users in the podTemplate, or assigned using a\n+\n+properties of the pod. Either way, all schedulers must be configured to understand the\n+same priorities (names and ordering). This could be done by making them constants in the\n+API, or using ConfigMap to configure the schedulers with the information. The advantage of\n+the former (at least making the names, if not the ordering, constants in the API) is that\n+it allows the API server to do validation (e.g. to catch mis-spelling).\n+\n+In the future, which priorities are usable for a given namespace and pods with certain\n+attributes may be configurable, similar to ResourceQuota, LimitRange, or security policy.\n+\n+Priority and resource QoS are indepedent.\n+\n+The priority we have described here might be used to prioritize the scheduling queue\n+(i.e. the order in which a scheduler examines pods in its scheduling loop), but the two\n+priority concepts do not have to be connected. It is somewhat logical to tie them\n+together, since a higher priority genreally indicates that a pod is more urgent to get\n+running. Also, scheduling low-priority pods before high-priority pods might lead to\n+avoidable preemptions if the high-priority pods end up preempting the low-priority pods\n+that were just scheduled.\n+\n+#### Relationship of priority to quota\n+\n+Of course, if the decision of what priority to give a pod is solely up to the user, then\n+users have no incentive to ever request any priority less than the maximum.  Thus\n+priority is intimately related to quota, in the sense that resource quotas must be\n+allocated on a per-priority-level basis (X amount of RAM at priority A, Y amount of RAM\n+at priority B, etc.). The \"guarantee\" that highest-priority pods will always be able to\n+schedule can only be achieved if the sum of the quotas at the top priority level is less\n+than or equal to the cluster capacity. This is analogous to QoS, where safety can only be\n+achieved if the sum of the limits of the top QoS level (\"Guaranteed\") is less than or\n+equal to the node capacity.\n+\n+The topic of how to allocate quota at different priority levels to achieve a desired\n+balance between utilization and probability of schedulability is an extremely complex\n+topic that is outside the scope of this document. For example, resource fragmentation and\n+RequiredDuringScheduling node and pod affinity and anti-affinity means that even if the\n+sum of the quotas at the top priority level is less than or equal to the total aggregate\n+capacity of the cluster, some pods at the top priority level might still go pending. In\n+general, priority provdes a *probabilistic* guarantees of pod schedulability in the face\n+of overcommitment, by allowing prioritization of which pods should be allowed to run pods\n+when demand for cluster resources exceeds supply.\n+\n+### Disruption budget\n+\n+While priority can protect pods from one source of disruption (preemption by a\n+lower-priority pod), *disruption budgets* limit disruptions from all Kubernetes-initiated\n+causes, including preemption by an equal or higher-priority pod, or being evicted to\n+achieve other rescheduling goals. In particular, each pod is optionally associated with a\n+\"disruption budget,\" a new API resource that limits Kubernetes-initiated terminations\n+across a set of pods (e.g. the pods of a particular Service might all point to the same\n+disruption budget object), regardless of cause. Initially we expect disruption budget\n+(e.g. `DisruptionBudgetSpec`) to consist of\n+* a rate limit on disruptions (preemption and other evictions) across the corresponding\n+  set of pods, e.g. no more than one disruption per hour across the pods of a particular Service\n+* a minimum number of pods that must be up simultaneously (sometimes called \"shard\n+  strength\") (of course this can also be expressed as the inverse, i.e. the number of\n+  pods of the collection that can be down simultaneously)\n+\n+The second item merits a bit more explanation. One use case is to specify a quorum size,\n+e.g. to ensure that at least 3 replicas of a quorum-based service with 5 replicas are up\n+at the same time. In practice, a service should always create enough replicas to survive\n+at least one planned and one unplanned outage. So in our quorum example, we would specify\n+that at least 4 replicas must be up at the same time; this allows for one intentional\n+disruption (bringing the number of live replicas down from 5 to 4 and consuming one unit\n+of shard strength budget) and one unplanned disruption (bringing the number of live\n+replicas down from 4 to 3) while still maintaining a quorum. Shard strength is also\n+useful for simpler replicated services; for example, you might not want more than 10% of\n+your front-ends to be down at the same time, so as to avoid overloading the remaining\n+replicas.\n+\n+Initially, disruption budgets will be specified by the user. Thus as with priority,\n+disruption budgets need to be tied into quota, to prevent users from saying none of their\n+pods can every be disrupted. The exact way of expressing and enforcing this quota is TBD,\n+though a simple starting point would be to assign a default disruption budget based on\n+priority level (more liberal with decreasing priority). We also likely need a quota that\n+applies to Kubernetes *components*, to the limit the rate at which any one component is\n+allowed to consume disruption budget.\n+\n+Of course there should also be a `DisruptionBudgetStatus` that indicates the current\n+disruption rate that the collection of pods is experiencing, and the number of pods that\n+are up.\n+\n+For the purposes of disruption budget, a pod is considered to be disrupted as soon as its\n+graceful termination period starts.\n+\n+TBD: In addition to `PodSpec`, where do we store pointer to disruption budget\n+(podTemplate in controller that managed the pod?)? Do we auto-generate a disruption\n+budget (e.g. when instantiating a Service), or require the user to create it manually\n+before they create a controller? Which objects should return the disruption budget object\n+as part of the output on `kubectl get` other than (obviously) `kubectl get` for the\n+disruption budget itself?\n+\n+TODO: Clean up distinction between \"down due to voluntary action taken by Kubernetes\"\n+and \"down due to unplanned outage\" in spec and status.\n+\n+For now, there is nothing to prevent clients from circumventing the disruption budget\n+protections. Of course, clients that do this are not being \"good citizens.\" In the next\n+section we describe a mechanism that at least makes it easy for well-behaved clients to\n+obey the disruption budgets.\n+\n+See #12611 for additional discussion of disruption budgets.\n+\n+### /evict subresource and PreferAvoidPods\n+\n+Although we could put the responsibility for checking and updating disruption budgets\n+solely on the client, it is safer and more convenient if we implement that functionality\n+in the API server. Thus we will introduce a new `/evict` subresource on pod. It is similar to\n+today's \"delete\" on pod except\n+  * It will be rejected if the deletion would violate disruption budget. (See how\n+  Deployment handles failure of /rollback for ideas on how clients could handle failure\n+  of `/evict`.) There are two possible ways to implement this:\n+    * For the initial implementation, this will be accomplished by the API server just\n+  looking at the `DisruptionBudgetStatus` and seeing if the disruption would violate the\n+  `DisruptionBudgetSpec`. In this approach, we assume a disruption budget controller\n+  keeps the `DisruptionBudgetStatus` up-to-date by observing all pod deletions and\n+  creations in the cluster, so that an approved disruption is quickly reflected in the\n+  `DisruptionBudgetStatus`. Of course this approach does allow a race in which one or\n+  more additional disruptions could be approved before the first one is reflected in the\n+  `DisruptionBudgetStatus`.\n+    * Thus a subsequent implementation will have the API server explicitly debit the\n+  `DisruptionBudgetStatus` when it accepts an `/evict`. (There still needs to be a\n+  controller, to keep the shard strength status up-to-date when replacement pods are\n+  created after an eviction; the controller may also be necessary for the rate status\n+  depending on how rate is represented, e.g. adding tokens to a bucket at a fixed rate.)\n+  Once etcd support multi-object transactions (etcd v3), the debit and pod deletion will\n+  be placed in the same transaction.\n+    * Note: For the purposes of disruption budget, a pod is considered to be disrupted as soon as its\n+  graceful termination period starts (so when we say \"delete\" here we do not mean\n+  \"deleted from etcd\" but rather \"graceful termination period has started\").\n+  * It will allow clients to communicate additional parameters when they wish to delete a\n+  pod. (In the absence of the `/evict` subresource, we would have to create a pod-specific\n+  type analogous to `api.DeleteOptions`.)\n+\n+We will make `kubectl delete pod` use `/evict` by default, and require a command-line\n+flag to delete the pod directly.\n+\n+We will add to `NodeStatus` a bounded-sized list of signatures of pods that should avoid\n+that node (provisionally called `PreferAvoidPods`). One of the pieces of information\n+specified in the `/evict` subresource is whether the eviction should add the evicted\n+pod's signature to the corresponding node's `PreferAvoidPods`. Initially the pod\n+signature will be a\n+[controllerRef](https://github.com/kubernetes/kubernetes/issues/14961#issuecomment-183431648),\n+i.e. a reference to the pod's controller. Controllers are responsible for garbage\n+collecting, after some period of time, `PreferAvoidPods` that point to them, but the API\n+server will also enforce a bounded size on the list. All schedulers will have a\n+highest-weighted priority function that gives a node the worst priority if the pod it is\n+scheduling appears in that node's `PreferAvoidPods` list. Thus appearing in\n+`PreferAvoidPods` is similar to\n+[RequiredDuringScheduling node anti-affinity](../../docs/user-guide/node-selection/README.md)\n+but it takes precedence over all other priority criteria and is not explicitly listed in\n+the `NodeAffinity` of the pod.\n+\n+`PreferAvoidPods` is useful for the \"moving a running pod off of a node from which it is\n+receiving poor service\" use case, as it reduces the chance that the replacement pod will\n+end up on the same node (keep in mind that most of those cases are situations that the\n+scheduler does not have explicit priority functions for, for example it cannot know in\n+advance that a pod will be starved). Also, though we do not intend to implement any such\n+policies in the first version of the rescheduler, it is useful whenever the rescheduler evicts\n+two pods A and B with the intention of moving A into the space vacated by B (it prevents\n+B from rescheduling back into the space it vacated before A's scheduler has a chance to\n+reschedule A there). Note that these two uses are subtly different; in the first\n+case we want the avoidance to last a relatively long time, whereas in the second case we\n+may only need it to last until A schedules.\n+\n+See #20699 for more discussion.\n+\n+### Preemption mechanics\n+\n+Pod schedulers will decide and enact preemptions, subject to the priority and disruption\n+budget rules described earlier. (Though note that we currently do not have any mechanism\n+to prevent schedulers from bypassing either the priority or disruption budget rules.)\n+The scheduler does not concern itself with whether the evicted pod(s) can reschedule. The\n+eviction(s) use(s) the `/evict` subresource so that it is subject to the disruption\n+budget(s) of the victim(s), but it does not request to add the victim pod(s) to the\n+nodes' `PreferAvoidPods`.\n+\n+Evicting victim(s) and binding the pending pod that the evictions are intended to enable\n+to schedule, are not transactional. We expect the scheduler to issue the operations in\n+sequence, but it is still possible that another scheduler could schedule its pod in\n+between the eviction(s) and the binding, or that the set of pods running on the node in\n+question changed between the time the scheduler made its decision and the time it sent\n+the operations to the API server thereby causing the eviction(s) to be not sufficient to get the\n+pending pod to schedule. In general there are a number of race conditions that cannot be\n+avoided without (1) making the evictions and binding be part of a single transaction, and\n+(2) making the binding preconditioned on a version number that is associated with the\n+node and is incremented on every binding. We may or may not implement those mechanisms in\n+the future.\n+\n+Given a choice between a node where scheduling a pod requires preemption and one where it\n+does not, all other things being equal, a scheduler should choose the one where\n+preemption is not required. (TBD: Also, if the selected node does require preemption, the\n+scheduler should preempt lower-priority pods before higher-priority pods (e.g. if the\n+scheduler needs to free up 4 GB of RAM, and the node has two 2 GB low-priority pods and\n+one 4 GB high-priority pod, all of which have sufficient disruption budget, it should\n+preempt the two low-priority pods). This is debatable, since all have sufficient\n+disruption budget. But still better to err on the side of giving better disruption SLO to\n+higher-priority pods when possible?)\n+\n+Preemption victims must be given their termination grace period. One possible sequence\n+of events is\n+\n+1. The API server binds the preemptor to the node (i.e. sets `nodeName` on the\n+preempting pod) and sets `deletionTimestamp` on the victims\n+2. Kubelet sees that `deletionTimestamp` has been set on the victims; they enter their\n+graceful termination period\n+3. Kubelet sees the preempting pod. It runs the admission checks on the new pod\n+assuming all pods that are in their graceful termination period are gone and that\n+all pods that are in the waiting state (see (4)) are running.\n+4. If (3) fails, then the new pod is rejected. If (3) passes, then Kubelet holds the\n+new pod in a waiting state, and does not run it until the pod passes passes the\n+admission checks using the set of actually running pods.\n+\n+Note that there are a lot of details to be figured out here; above is just a very\n+hand-wavy sketch of one general approach that might work.\n+\n+See #22212 for additional discussion.\n+\n+### Node drain\n+\n+Node drain will be handled by one or more components not described in this document. They\n+will respect disruption budgets. Initially, we will just make `kubectl drain`\n+respect disruption budgets.  See #17393 for other discussion.\n+\n+### Rescheduler\n+\n+All rescheduling other than preemption and node drain will be decided and enacted by a\n+new component called the *rescheduler*. It runs continuously in the background, looking\n+for opportunities to move pods to better locations. It acts when the degree of\n+improvement meets some threshold and is allowed by the pod's disruption budget.  The\n+action is eviction of a pod using the `/evict` subresource, with the pod's signature\n+enqueued in the node's `PreferAvoidPods`. It does not force the pod to reschedule to any\n+particular node. Thus it is really an \"unscheduler\"; only in combination with the evicted\n+pod's scheduler, which schedules the replacement pod, do we get true \"rescheduling.\"  See\n+the \"Example use cases\" section earlier for some example use cases.\n+\n+The rescheduler is a best-effort service that makes no guarantees about how quickly (or\n+whether) it will resolve a suboptimal pod placement.\n+\n+The first version of the rescheduler will not take into consideration where or whether an\n+evicted pod will reschedule. The evicted pod may go pending, consuming one unit of the\n+corresponding shard strength disruption budget by one indefinitely. By using the `/evict`\n+subresource, the rescheduler ensures that an evicted pod has sufficient budget for the\n+evicted pod to go and stay pending.  We expect future versions of the rescheduler may be\n+linked with the \"mandatory\" predicate functions (currently, the ones that constitute the\n+Kubelet admission criteria), and will only evict if the rescheduler determines that the\n+pod can reschedule somewhere according to those criteria. (Note that this still does not\n+guarantee that the pod actually will be able to reschedule, for at least two reasons: (1)\n+the state of the cluster may change between the time the rescheduler evaluates it and\n+when the evicted pod's scheduler tries to schedule the replacement pod, and (2) the\n+evicted pod's scheduler may have additional predicate functions in addition to the\n+mandatory ones).\n+\n+The first version of the rescheduler will only implement two objectives: moving a pod\n+onto an under-utilized node, and moving a pod onto a node that meets more of the pod's\n+affinity/anti-affinity preferences than wherever it is currently running. (We assume that\n+nodes that are intentionally under-utilized, e.g. because they are being drained, are\n+marked unschedulable, thus the first objective will not cause the rescheduler to \"fight\"\n+a system that is draining nodes.)  We assume that all schedulers sufficiently weight the\n+priority functions for affinity/anti-affinity and avoiding very packed nodes,\n+otherwise evicted pods may not actually move onto a node that is better according to\n+the criteria that caused it to be evicted. (But note that in all cases it will move to a\n+node that is better according to the totality of its scheduler's priority functions,\n+except in the case where the node where it was already running was the only node\n+where it can run.) As a general rule, the rescheduler should only act when it sees\n+particularly bad situations, since (1) an eviction for a marginal improvement is likely\n+not worth the disruption--just because there is sufficient budget for an eviction doesn't\n+mean an eviction is painless to the application, and (2) rescheduling the pod might not\n+actually mitigate the identified problem if it is minor enough that other scheduling\n+factors dominate the decision of where the replacement pod is scheduled.\n+\n+We assume schedulers' priority functions are at least vaguely aligned with the\n+rescheduler's policies; otherwise the rescheduler will never accomplish anything useful,\n+given that it relies on the schedulers to actually reschedule the evicted pods. (Even if\n+the rescheduler acted as a scheduler, explicitly rebinding evicted pods, we'd still want\n+this to be true, to prevent the schedulers and rescheduler from \"fighting\" one another.)\n+","path":"docs/proposals/rescheduling.md","position":440,"original_position":440,"commit_id":"a543cfe3de6872f60c7d0a72e084082603cbcf72","original_commit_id":"a543cfe3de6872f60c7d0a72e084082603cbcf72","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"body":"I agree that we might want some way to force the evicted pod to the front of the scheduler queue, independently of its priority.","created_at":"2016-03-09T01:21:21Z","updated_at":"2016-03-09T01:21:21Z","html_url":"https://github.com/kubernetes/kubernetes/pull/22217#discussion_r55458921","pull_request_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/55458921"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/22217#discussion_r55458921"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217","id":61159296,"html_url":"https://github.com/kubernetes/kubernetes/pull/22217","diff_url":"https://github.com/kubernetes/kubernetes/pull/22217.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/22217.patch","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22217","number":22217,"state":"open","locked":false,"title":"[WIP/RFC] Rescheduling in Kubernetes design proposal","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"body":"Proposal by @bgrant0607 and @davidopp (and inspired by years of discussion and experience from folks who worked on Borg and Omega).\r\n\r\nThis doc is a proposal for a set of inter-related concepts related to \"rescheduling\" -- that is, \"moving\" an already-running pod to a new node in order to improve where it is running. (Specific concepts discussed are priority, preemption, disruption budget, quota, `/evict` subresource, and rescheduler.)\r\n\r\nFeedback on the proposal is very welcome. For now, please stick to comments about the design, not spelling, punctuation, grammar, broken links, etc., so we can keep the doc uncluttered enough to make it easy for folks to comment on the more important things. \r\n\r\nref/ #22054 #18724 #19080 #12611 #20699 #17393 #12140 #22212\r\n\r\n@HaiyangDING @mqliang @derekwaynecarr @kubernetes/sig-scheduling @kubernetes/huawei @timothysc @mml @dchen1107\r\n","created_at":"2016-02-29T22:42:30Z","updated_at":"2016-03-09T01:21:21Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0792d0c5ba3b0e2a578241219520d0210d54ae0a","assignee":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217/commits","review_comments_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217/comments","review_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22217/comments","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/a543cfe3de6872f60c7d0a72e084082603cbcf72","head":{"label":"davidopp:rescheduling","ref":"rescheduling","sha":"a543cfe3de6872f60c7d0a72e084082603cbcf72","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"repo":{"id":42850455,"name":"kubernetes","full_name":"davidopp/kubernetes","owner":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/davidopp/kubernetes","description":"Container Cluster Manager from Google","fork":true,"url":"https://api.github.com/repos/davidopp/kubernetes","forks_url":"https://api.github.com/repos/davidopp/kubernetes/forks","keys_url":"https://api.github.com/repos/davidopp/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/davidopp/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/davidopp/kubernetes/teams","hooks_url":"https://api.github.com/repos/davidopp/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/davidopp/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/davidopp/kubernetes/events","assignees_url":"https://api.github.com/repos/davidopp/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/davidopp/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/davidopp/kubernetes/tags","blobs_url":"https://api.github.com/repos/davidopp/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/davidopp/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/davidopp/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/davidopp/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/davidopp/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/davidopp/kubernetes/languages","stargazers_url":"https://api.github.com/repos/davidopp/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/davidopp/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/davidopp/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/davidopp/kubernetes/subscription","commits_url":"https://api.github.com/repos/davidopp/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/davidopp/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/davidopp/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/davidopp/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/davidopp/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/davidopp/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/davidopp/kubernetes/merges","archive_url":"https://api.github.com/repos/davidopp/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/davidopp/kubernetes/downloads","issues_url":"https://api.github.com/repos/davidopp/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/davidopp/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/davidopp/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/davidopp/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/davidopp/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/davidopp/kubernetes/releases{/id}","deployments_url":"https://api.github.com/repos/davidopp/kubernetes/deployments","created_at":"2015-09-21T07:23:57Z","updated_at":"2015-09-21T07:24:06Z","pushed_at":"2016-03-07T19:32:23Z","git_url":"git://github.com/davidopp/kubernetes.git","ssh_url":"git@github.com:davidopp/kubernetes.git","clone_url":"https://github.com/davidopp/kubernetes.git","svn_url":"https://github.com/davidopp/kubernetes","homepage":"http://kubernetes.io","size":176266,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"83d74dd9528e424c503c05795fe685ca8404630b","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"kubernetes/kubernetes","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/kubernetes/kubernetes","forks_url":"https://api.github.com/repos/kubernetes/kubernetes/forks","keys_url":"https://api.github.com/repos/kubernetes/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/kubernetes/teams","hooks_url":"https://api.github.com/repos/kubernetes/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/kubernetes/events","assignees_url":"https://api.github.com/repos/kubernetes/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/kubernetes/tags","blobs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/kubernetes/languages","stargazers_url":"https://api.github.com/repos/kubernetes/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/kubernetes/subscription","commits_url":"https://api.github.com/repos/kubernetes/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/kubernetes/merges","archive_url":"https://api.github.com/repos/kubernetes/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/kubernetes/downloads","issues_url":"https://api.github.com/repos/kubernetes/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/kubernetes/releases{/id}","deployments_url":"https://api.github.com/repos/kubernetes/kubernetes/deployments","created_at":"2014-06-06T22:56:04Z","updated_at":"2016-03-08T23:20:39Z","pushed_at":"2016-03-09T01:18:44Z","git_url":"git://github.com/kubernetes/kubernetes.git","ssh_url":"git@github.com:kubernetes/kubernetes.git","clone_url":"https://github.com/kubernetes/kubernetes.git","svn_url":"https://github.com/kubernetes/kubernetes","homepage":"http://kubernetes.io","size":336467,"stargazers_count":13097,"watchers_count":13097,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3766,"mirror_url":null,"open_issues_count":2977,"forks":3766,"open_issues":2977,"watchers":13097,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/22217"},"issue":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/22217"},"comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/22217/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/22217/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/kubernetes/statuses/a543cfe3de6872f60c7d0a72e084082603cbcf72"}}}}