{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22545","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22545/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22545/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22545/events","html_url":"https://github.com/kubernetes/kubernetes/issues/22545","id":138563097,"number":22545,"title":"Daemonsets are forever","user":{"login":"therc","id":13481082,"avatar_url":"https://avatars.githubusercontent.com/u/13481082?v=3","gravatar_id":"","url":"https://api.github.com/users/therc","html_url":"https://github.com/therc","followers_url":"https://api.github.com/users/therc/followers","following_url":"https://api.github.com/users/therc/following{/other_user}","gists_url":"https://api.github.com/users/therc/gists{/gist_id}","starred_url":"https://api.github.com/users/therc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/therc/subscriptions","organizations_url":"https://api.github.com/users/therc/orgs","repos_url":"https://api.github.com/users/therc/repos","events_url":"https://api.github.com/users/therc/events{/privacy}","received_events_url":"https://api.github.com/users/therc/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P2","name":"priority/P2","color":"fbca04"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/ux","name":"team/ux","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/17","html_url":"https://github.com/kubernetes/kubernetes/milestones/next-candidate","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/17/labels","id":1510379,"number":17,"title":"next-candidate","description":"Candidates for the next milestone","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":259,"closed_issues":11,"state":"open","created_at":"2016-01-16T17:58:20Z","updated_at":"2016-03-04T23:12:32Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-03-04T19:24:32Z","updated_at":"2016-03-04T23:23:48Z","closed_at":null,"body":"Two nodes went away as I updated the instances, but they still appear and, more importantly/confusingly, their daemonset pods are still in Running state. \r\n\r\n```\r\n$ fleetctl list-machines\r\nMACHINE         IP              METADATA\r\n729947b9...     10.0.12.180     role=node\r\na0883347...     10.0.8.4        instance=m0,role=master\r\n$ kubectl get nodes\r\nNAME                          STATUS                     AGE\r\nip-10-0-10-222.ec2.internal   NotReady                   2d\r\nip-10-0-12-147.ec2.internal   NotReady                   1d\r\nip-10-0-12-180.ec2.internal   Ready                      17m\r\nip-10-0-8-4.ec2.internal      Ready,SchedulingDisabled   2d\r\n$ kubectl get pods --all-namespaces\r\nNAMESPACE     NAME                    READY     STATUS        RESTARTS   AGE\r\nkube-system   fluentd-logging-2gxb1   1/1       Running       0          1d\r\nkube-system   fluentd-logging-33y2f   1/1       Running       0          1d\r\nkube-system   fluentd-logging-b1yuv   1/1       Running       0          17m\r\nkube-system   fluentd-logging-o9hvp   1/1       Running       0          1d\r\n```\r\n\r\nThere are only two instances and thus two daemons up, not four.\r\nI understand the rationale for not killing the pods outright, but is there a timeout after which they change state or perhaps the nodes do?\r\n\r\nMy concern is twofold:\r\n 1. `Running` is a lie. Things will be confusing for monitoring/accounting/troubleshooting purposes.\r\n 1. State about pods and nodes is kept around for a long time. Imagine someone using DaemonSets on a pool of EC2 spot instances..."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/192518017","html_url":"https://github.com/kubernetes/kubernetes/issues/22545#issuecomment-192518017","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/22545","id":192518017,"user":{"login":"mikedanese","id":1787169,"avatar_url":"https://avatars.githubusercontent.com/u/1787169?v=3","gravatar_id":"","url":"https://api.github.com/users/mikedanese","html_url":"https://github.com/mikedanese","followers_url":"https://api.github.com/users/mikedanese/followers","following_url":"https://api.github.com/users/mikedanese/following{/other_user}","gists_url":"https://api.github.com/users/mikedanese/gists{/gist_id}","starred_url":"https://api.github.com/users/mikedanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikedanese/subscriptions","organizations_url":"https://api.github.com/users/mikedanese/orgs","repos_url":"https://api.github.com/users/mikedanese/repos","events_url":"https://api.github.com/users/mikedanese/events{/privacy}","received_events_url":"https://api.github.com/users/mikedanese/received_events","type":"User","site_admin":false},"created_at":"2016-03-04T23:23:48Z","updated_at":"2016-03-04T23:23:48Z","body":"@therc if you kubectl delete node {node} for the nodes they are on, they will go away i think. only kubelet is allowed to say whether the nodes are running or not. Hopefully the pods have the not ready condition :). Without the nodes being deleted from the api, the master thinks there is a network partion and has no way to know whether the pods are running or not. I think surfacing Unkown, condition in the api would be a good idea here.\r\n\r\n@mml @davidopp this is the \"no evict\" stuff we added to the node controller."}}