{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62928","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62928/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62928/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62928/events","html_url":"https://github.com/kubernetes/kubernetes/issues/62928","id":316461147,"node_id":"MDU6SXNzdWUzMTY0NjExNDc=","number":62928,"title":"kube-controller-manager stopped syncing daemonset","user":{"login":"ryanmcnamara","id":3452559,"node_id":"MDQ6VXNlcjM0NTI1NTk=","avatar_url":"https://avatars3.githubusercontent.com/u/3452559?v=4","gravatar_id":"","url":"https://api.github.com/users/ryanmcnamara","html_url":"https://github.com/ryanmcnamara","followers_url":"https://api.github.com/users/ryanmcnamara/followers","following_url":"https://api.github.com/users/ryanmcnamara/following{/other_user}","gists_url":"https://api.github.com/users/ryanmcnamara/gists{/gist_id}","starred_url":"https://api.github.com/users/ryanmcnamara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryanmcnamara/subscriptions","organizations_url":"https://api.github.com/users/ryanmcnamara/orgs","repos_url":"https://api.github.com/users/ryanmcnamara/repos","events_url":"https://api.github.com/users/ryanmcnamara/events{/privacy}","received_events_url":"https://api.github.com/users/ryanmcnamara/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":778118404,"node_id":"MDU6TGFiZWw3NzgxMTg0MDQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lifecycle/stale","name":"lifecycle/stale","color":"795548","default":false},{"id":404091735,"node_id":"MDU6TGFiZWw0MDQwOTE3MzU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/apps","name":"sig/apps","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-04-21T04:39:10Z","updated_at":"2018-11-17T22:44:54Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"/kind bug\r\n\r\nkube-controller-manager stopped syncing a daemonset. I noticed this when I observed that I had several k8s nodes that I expected pods from the deamonset to be running on. I restarted the master kube-controller-manager (we are running HA kubernetes) and shortly after (within a minute) all kubelets had begun running the pod. Exact timeline:\r\n\r\nI0418 18:49:52.817998 - daemonset starts syncing\r\nI0418 19:51:46.278384 - daemonset stops syncing\r\n\r\nHere are the redacted / reduced kube-controller-manager from the duration that confirm the above timeline.\r\nhttps://gist.github.com/ryanmcnamara/9caa9272feeb4822daa2c2683a19ee1a\r\n\r\nto get these logs I did the following starting with the raw kube-controller-manager logs\r\n1. Only keep lines containing \"daemon\"\r\n1. Deleted lines containing other deamonsets not in question here\r\n1. Remove lines containing \"error storing status\" and \"the object has been modified\", these lines contained the exact daemonset config and I don't think are relevant\r\n\r\nLike I said, restarting the kube-controller-manager immediately fixed this issue.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nI'm not sure how to repro this, I'm hoping to get guidance about either\r\n1. What to look for in the logs\r\n1. Other advice on how to debug this\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`): 1.9.3\r\n- Cloud provider or hardware configuration: aws"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/439653079","html_url":"https://github.com/kubernetes/kubernetes/issues/62928#issuecomment-439653079","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62928","id":439653079,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTY1MzA3OQ==","user":{"login":"fejta-bot","id":30488982,"node_id":"MDQ6VXNlcjMwNDg4OTgy","avatar_url":"https://avatars1.githubusercontent.com/u/30488982?v=4","gravatar_id":"","url":"https://api.github.com/users/fejta-bot","html_url":"https://github.com/fejta-bot","followers_url":"https://api.github.com/users/fejta-bot/followers","following_url":"https://api.github.com/users/fejta-bot/following{/other_user}","gists_url":"https://api.github.com/users/fejta-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/fejta-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fejta-bot/subscriptions","organizations_url":"https://api.github.com/users/fejta-bot/orgs","repos_url":"https://api.github.com/users/fejta-bot/repos","events_url":"https://api.github.com/users/fejta-bot/events{/privacy}","received_events_url":"https://api.github.com/users/fejta-bot/received_events","type":"User","site_admin":false},"created_at":"2018-11-17T22:44:54Z","updated_at":"2018-11-17T22:44:54Z","author_association":"NONE","body":"Stale issues rot after 30d of inactivity.\nMark the issue as fresh with `/remove-lifecycle rotten`.\nRotten issues close after an additional 30d of inactivity.\n\nIf this issue is safe to close now please do so with `/close`.\n\nSend feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).\n/lifecycle rotten"}}
{"actor":{"display_login":"fejta-bot"}}