{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31762","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31762/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31762/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31762/events","html_url":"https://github.com/kubernetes/kubernetes/issues/31762","id":174163573,"number":31762,"title":"PetSet fencing","user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/stateful-apps","name":"area/stateful-apps","color":"ff6060"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"assignees":[{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.5","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23/labels","id":1945978,"number":23,"title":"v1.5","description":"","creator":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"open_issues":54,"closed_issues":1,"state":"open","created_at":"2016-08-16T21:47:28Z","updated_at":"2016-08-31T01:25:36Z","due_on":"2016-12-09T00:00:00Z","closed_at":null},"comments":0,"created_at":"2016-08-31T01:25:36Z","updated_at":"2016-08-31T01:25:36Z","closed_at":null,"body":"Jotting down some thoughts so we can start fleshing it out in time for 1.5.\r\n\r\n## Problem \r\n\r\nCurrently petset uses ns/name as an implicit lock. So there are situation where we might: \r\n* deleted a pod from the apiserver\r\n* but the pod is still alive on the node\r\n* and the petset controller has created another pod with the same name on another node\r\n* and kube-dns has re-programmed dns to point pet-0 at this new ip \r\n* but some clients are pointing at pet-0(2) and some at pet-0(1) because they haven't resolved dns. \r\n\r\nTo prevent such situations, we need to ensure that exactly one instances of an application occupies a given petset index. Since this is not a new problem in distributed systems, one option is to completely defer to the application layer to implement a consistency protocol. In reality this places a pretty big burden on the app.\r\n\r\n## Proposal \r\nSo I'm proposing the following fencing guards: \r\n* Don't delete pets from node controller (or make this conditional)\r\n* Use a RWO volume for fencing\r\n* Take a lease via apiserver annotations\r\n\r\nThe first eliminates ONLY the most common cause of duplicity. \r\nThe second only works if you trust the storage layer, or actually need pet storage.\r\nThe third needs to be our last line of defense, and happen at 2 points of the pet lifecycle: \r\n  1. Prevent PetSet controller from creating pet-0(2) based on the lock annotation\r\n  2. Prevent pet-0(2) from ever starting by blockig in an init container\r\n\r\nNone of these require changes to the petset resources IMO, unless we decide to add the locks to the petset.\r\n\r\n## Example scenario\r\nTo break down some cases, say we do the following:   \r\n\r\n1. Pipe the petset name as an env var into all pets\r\n2. Each pet writes an annotation of the form {$POD_NAME: $POD_UUID} in the petset\r\n3. The pet deletes the annotation in terminationGrace/postStop\r\n\r\nAssuming we have a node controller or kubelet bug that leads to node controller actually deleting pet-0, or user deletes pet-0 but kubelet fails to kill container, the petset controller will not create pet-0(2) because of the annotation.\r\n\r\nAssuming we also have a petset controller bug, so petset-0(2) gets created: \r\n* pet-0(2) comes up and reads {pet-0:different-uuid}, crashloops\r\n* kubelet deletes pet-0 (either because partition recovers, or it recovers from its own bug)\r\n* pet-0 erases annotation within terminationGrace\r\n* pet-0(2) starts up and writes its annotation\r\n\r\nThe impact is a loss of availability during partition, and if there's a second bug that prevents terminationGrace on pet-0, pet-0(2) will remain in crashloop till human erases annotation.\r\n\r\n\r\n## Food for thought     \r\n\r\n1. Do we need a third layer of defense in the form of a sidecar that commits suicide if the annotation is set to something else?\r\n2. We should probably allow nodecontroller deletion gate to be opt in\r\n3. This adds a coupling between pets and apiserver that I dislike\r\n\r\n@kubernetes/sig-apps @smarterclayton @erictune "}}
{"actor":{"display_login":"bprashanth"}}