{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/45419","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/45419/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/45419/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/45419/events","html_url":"https://github.com/kubernetes/kubernetes/issues/45419","id":226598989,"node_id":"MDU6SXNzdWUyMjY1OTg5ODk=","number":45419,"title":"Node flapping between Ready/NotReady with PLEG issues","user":{"login":"deitch","id":282526,"node_id":"MDQ6VXNlcjI4MjUyNg==","avatar_url":"https://avatars3.githubusercontent.com/u/282526?v=4","gravatar_id":"","url":"https://api.github.com/users/deitch","html_url":"https://github.com/deitch","followers_url":"https://api.github.com/users/deitch/followers","following_url":"https://api.github.com/users/deitch/following{/other_user}","gists_url":"https://api.github.com/users/deitch/gists{/gist_id}","starred_url":"https://api.github.com/users/deitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deitch/subscriptions","organizations_url":"https://api.github.com/users/deitch/orgs","repos_url":"https://api.github.com/users/deitch/repos","events_url":"https://api.github.com/users/deitch/events{/privacy}","received_events_url":"https://api.github.com/users/deitch/received_events","type":"User","site_admin":false},"labels":[{"id":173493665,"node_id":"MDU6TGFiZWwxNzM0OTM2NjU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":108,"created_at":"2017-05-05T14:55:46Z","updated_at":"2018-06-13T22:10:34Z","closed_at":null,"author_association":"NONE","body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n\r\n**Is this a request for help?** No\r\n\r\n\r\n**What keywords did you search in Kubernetes issues before filing this one?** (If you have found any duplicates, you should instead reply there.): PLEG NotReady kubelet\r\n\r\n---\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?** Bug\r\n\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**Kubernetes version** (use `kubectl version`): 1.6.2\r\n\r\n\r\n**Environment**:\r\n- **Cloud provider or hardware configuration**: CoreOS on AWS\r\n- **OS** (e.g. from /etc/os-release):CoreOS 1353.7.0\r\n- **Kernel** (e.g. `uname -a`): 4.9.24-coreos\r\n- **Install tools**:\r\n- **Others**:\r\n\r\n\r\n**What happened**:\r\n\r\nI have a 3-worker cluster. Two and sometimes all three nodes keep dropping into `NotReady`with the following messages in `journalctl -u kubelet`:\r\n\r\n```\r\nMay 05 13:59:56 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 13:59:56.872880    2858 kubelet_node_status.go:379] Recording NodeNotReady event message for node ip-10-50-20-208.ec2.internal\r\nMay 05 13:59:56 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 13:59:56.872908    2858 kubelet_node_status.go:682] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2017-05-05 13:59:56.872865742 +0000 UTC LastTransitionTime:2017-05-05 13:59:56.872865742 +0000 UTC Reason:KubeletNotReady Message:PLEG is not healthy: pleg was last seen active 3m7.629592089s ago; threshold is 3m0s}\r\nMay 05 14:07:57 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:07:57.598132    2858 kubelet_node_status.go:379] Recording NodeNotReady event message for node ip-10-50-20-208.ec2.internal\r\nMay 05 14:07:57 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:07:57.598162    2858 kubelet_node_status.go:682] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2017-05-05 14:07:57.598117026 +0000 UTC LastTransitionTime:2017-05-05 14:07:57.598117026 +0000 UTC Reason:KubeletNotReady Message:PLEG is not healthy: pleg was last seen active 3m7.346983738s ago; threshold is 3m0s}\r\nMay 05 14:17:58 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:17:58.536101    2858 kubelet_node_status.go:379] Recording NodeNotReady event message for node ip-10-50-20-208.ec2.internal\r\nMay 05 14:17:58 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:17:58.536134    2858 kubelet_node_status.go:682] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2017-05-05 14:17:58.536086605 +0000 UTC LastTransitionTime:2017-05-05 14:17:58.536086605 +0000 UTC Reason:KubeletNotReady Message:PLEG is not healthy: pleg was last seen active 3m7.275467289s ago; threshold is 3m0s}\r\nMay 05 14:29:59 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:29:59.648922    2858 kubelet_node_status.go:379] Recording NodeNotReady event message for node ip-10-50-20-208.ec2.internal\r\nMay 05 14:29:59 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:29:59.648952    2858 kubelet_node_status.go:682] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2017-05-05 14:29:59.648910669 +0000 UTC LastTransitionTime:2017-05-05 14:29:59.648910669 +0000 UTC Reason:KubeletNotReady Message:PLEG is not healthy: pleg was last seen active 3m7.377520804s ago; threshold is 3m0s}\r\nMay 05 14:44:00 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:44:00.938266    2858 kubelet_node_status.go:379] Recording NodeNotReady event message for node ip-10-50-20-208.ec2.internal\r\nMay 05 14:44:00 ip-10-50-20-208.ec2.internal kubelet[2858]: I0505 14:44:00.938297    2858 kubelet_node_status.go:682] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2017-05-05 14:44:00.938251338 +0000 UTC LastTransitionTime:2017-05-05 14:44:00.938251338 +0000 UTC Reason:KubeletNotReady Message:PLEG is not healthy: pleg was last seen active 3m7.654775919s ago; threshold is 3m0s}\r\n```\r\n\r\ndocker daemon is fine (local `docker ps`, `docker images`, etc. all work and respond immediately). \r\n\r\nusing weave networking installed via `kubectl apply -f https://git.io/weave-kube-1.6`\r\n\r\n**What you expected to happen**:\r\n\r\nNodes to be ready.\r\n\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\n\r\nWish I knew how!\r\n\r\n\r\n**Anything else we need to know**:\r\n\r\nAll of the nodes (workers and masters) on same private subnet with NAT gateway to Internet. Workers in security group that allows unlimited access (all ports) from masters security group; masters allow all ports from same subnet. proxy is running on workers; apiserver, controller-manager, scheduler on masters. \r\n\r\n`kubectl logs` and `kubectl exec` always hang, even when run from the master itself (or from outside). "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/397104022","html_url":"https://github.com/kubernetes/kubernetes/issues/45419#issuecomment-397104022","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/45419","id":397104022,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzEwNDAyMg==","user":{"login":"phspagiari","id":968339,"node_id":"MDQ6VXNlcjk2ODMzOQ==","avatar_url":"https://avatars0.githubusercontent.com/u/968339?v=4","gravatar_id":"","url":"https://api.github.com/users/phspagiari","html_url":"https://github.com/phspagiari","followers_url":"https://api.github.com/users/phspagiari/followers","following_url":"https://api.github.com/users/phspagiari/following{/other_user}","gists_url":"https://api.github.com/users/phspagiari/gists{/gist_id}","starred_url":"https://api.github.com/users/phspagiari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/phspagiari/subscriptions","organizations_url":"https://api.github.com/users/phspagiari/orgs","repos_url":"https://api.github.com/users/phspagiari/repos","events_url":"https://api.github.com/users/phspagiari/events{/privacy}","received_events_url":"https://api.github.com/users/phspagiari/received_events","type":"User","site_admin":false},"created_at":"2018-06-13T22:10:34Z","updated_at":"2018-06-13T22:10:34Z","author_association":"NONE","body":"+1\r\nKubernetes: 1.7.16\r\nDocker: 17.12.1-ce\r\nOS: CoreOS 1688.5.3\r\nKernel: 4.14.32-coreos\r\nCNI: Calico (v2.6.7)\r\nKops v1.9.1"}}
{"actor":{"display_login":"phspagiari"}}