{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63858","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63858/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63858/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63858/events","html_url":"https://github.com/kubernetes/kubernetes/issues/63858","id":323188949,"number":63858,"title":"Bug: ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)","user":{"login":"KIVagant","id":517501,"avatar_url":"https://avatars2.githubusercontent.com/u/517501?v=4","gravatar_id":"","url":"https://api.github.com/users/KIVagant","html_url":"https://github.com/KIVagant","followers_url":"https://api.github.com/users/KIVagant/followers","following_url":"https://api.github.com/users/KIVagant/following{/other_user}","gists_url":"https://api.github.com/users/KIVagant/gists{/gist_id}","starred_url":"https://api.github.com/users/KIVagant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KIVagant/subscriptions","organizations_url":"https://api.github.com/users/KIVagant/orgs","repos_url":"https://api.github.com/users/KIVagant/repos","events_url":"https://api.github.com/users/KIVagant/events{/privacy}","received_events_url":"https://api.github.com/users/KIVagant/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":173493665,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-05-15T12:07:57Z","updated_at":"2018-05-15T21:28:12Z","closed_at":null,"author_association":"NONE","body":"**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n\r\n/kind bug\r\n\r\n**What happened**:\r\n\r\n- Nodes in `NotReady` status\r\n- Pods in `Unknown` status\r\n\r\n**What you expected to happen**:\r\n\r\n\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\n\r\n**Anything else we need to know?**:\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.2\", GitCommit:\"81753b10df112992bf51bbc2c2f85208aad78335\", GitTreeState:\"clean\", BuildDate:\"2018-04-27T09:22:21Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"10\", GitVersion:\"v1.10.0\", GitCommit:\"fc32d2f3698e36b93322a3465f63a14e9f0eaead\", GitTreeState:\"clean\", BuildDate:\"2018-03-26T16:44:10Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n- Cloud provider or hardware configuration: AWS\r\n- OS (e.g. from /etc/os-release): Debian 8 (jessie)\r\n- Kernel (e.g. `uname -a`): `Linux ... 4.4.115-k8s #1 SMP Thu Feb 8 15:37:40 UTC 2018 x86_64 GNU/Linux`\r\n- Install tools: kops\r\n- Others:\r\n    - Logs:\r\n\r\n```\r\n==> /var/log/daemon.log <==\r\nMay 15 12:02:19 ip-10-133-93-99 kubelet[2524]: E0515 12:02:19.150508    2524 remote_runtime.go:262] ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)\r\n\r\nMay 15 12:02:19 ip-10-133-93-99 kubelet[2524]: E0515 12:02:19.150586    2524 kuberuntime_container.go:323] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)\r\n\r\nMay 15 12:02:19 ip-10-133-93-99 kubelet[2524]: E0515 12:02:19.150613    2524 generic.go:197] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)\r\n\r\n\r\n==> /var/log/syslog <==\r\nMay 15 12:02:21 ip-10-133-93-99 kubelet[2524]: E0515 12:02:21.554728    2524 remote_runtime.go:262] ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)\r\n\r\nMay 15 12:02:21 ip-10-133-93-99 kubelet[2524]: E0515 12:02:21.554842    2524 kuberuntime_container.go:323] getKubeletContainers failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)\r\n\r\nMay 15 12:02:21 ip-10-133-93-99 kubelet[2524]: E0515 12:02:21.554873    2524 generic.go:197] GenericPLEG: Unable to retrieve pods: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4195017 vs. 4194304)\r\n\r\nMay 15 12:02:21 ip-10-133-93-99 kubelet[2524]: I0515 12:02:21.788754    2524 kubelet.go:1794] skipping pod synchronization - [PLEG is not healthy: pleg was last seen active 1h3m27.338264092s ago; threshold is 3m0s]\r\n```\r\n\r\n- Possible prerequisites: a lot of jobs (1657) that failed (state: Error) because of internal problems\r\n- Cluster size: 3 masters, 4 worker nodes\r\n- **Workaround:** how to get the node back\r\n\r\n```\r\ndocker system prune\r\n...\r\nservice docker restart && service kubelet restart\r\n```\r\n\r\n- Possible related to https://github.com/kubernetes/kubernetes/issues/51099\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/389319322","html_url":"https://github.com/kubernetes/kubernetes/issues/63858#issuecomment-389319322","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63858","id":389319322,"user":{"login":"mcluseau","id":3868856,"avatar_url":"https://avatars1.githubusercontent.com/u/3868856?v=4","gravatar_id":"","url":"https://api.github.com/users/mcluseau","html_url":"https://github.com/mcluseau","followers_url":"https://api.github.com/users/mcluseau/followers","following_url":"https://api.github.com/users/mcluseau/following{/other_user}","gists_url":"https://api.github.com/users/mcluseau/gists{/gist_id}","starred_url":"https://api.github.com/users/mcluseau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcluseau/subscriptions","organizations_url":"https://api.github.com/users/mcluseau/orgs","repos_url":"https://api.github.com/users/mcluseau/repos","events_url":"https://api.github.com/users/mcluseau/events{/privacy}","received_events_url":"https://api.github.com/users/mcluseau/received_events","type":"User","site_admin":false},"created_at":"2018-05-15T21:28:12Z","updated_at":"2018-05-15T21:28:12Z","author_association":"CONTRIBUTOR","body":"Hitting the same bug here (Kubernetes 1.10.2 with docker 17.12.1). The node has 8309 containers, most of them exited, created by failing jobs."}}
{"actor":{"display_login":"mcluseau"}}