{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/weave/issues/3695","repository_url":"https://api.github.com/repos/weaveworks/weave","labels_url":"https://api.github.com/repos/weaveworks/weave/issues/3695/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/weave/issues/3695/comments","events_url":"https://api.github.com/repos/weaveworks/weave/issues/3695/events","html_url":"https://github.com/weaveworks/weave/issues/3695","id":488247323,"node_id":"MDU6SXNzdWU0ODgyNDczMjM=","number":3695,"title":"Node loses pod connectivity after OOM","user":{"login":"SeMeKh","id":1090449,"node_id":"MDQ6VXNlcjEwOTA0NDk=","avatar_url":"https://avatars0.githubusercontent.com/u/1090449?v=4","gravatar_id":"","url":"https://api.github.com/users/SeMeKh","html_url":"https://github.com/SeMeKh","followers_url":"https://api.github.com/users/SeMeKh/followers","following_url":"https://api.github.com/users/SeMeKh/following{/other_user}","gists_url":"https://api.github.com/users/SeMeKh/gists{/gist_id}","starred_url":"https://api.github.com/users/SeMeKh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SeMeKh/subscriptions","organizations_url":"https://api.github.com/users/SeMeKh/orgs","repos_url":"https://api.github.com/users/SeMeKh/repos","events_url":"https://api.github.com/users/SeMeKh/events{/privacy}","received_events_url":"https://api.github.com/users/SeMeKh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2019-09-02T15:44:06Z","updated_at":"2019-10-09T07:38:56Z","closed_at":null,"author_association":"NONE","body":"In random situations of OOMKiller getting triggered, after the node is back up again (i.e. in `Ready` state) the node loses its pod connectivity.\r\nDeleting the weave pod (and consequently it getting recreated) makes the issue go away.\r\n\r\n## What you expected to happen?\r\nI expected the node to eventually recover from the OOM, and/or report its state as `NotReady` if it hasn't.\r\n\r\n## What happened?\r\nThe node reports its network state as ready, but one can not access pod IPs from that node or pods running on it.\r\n\r\n## How to reproduce it?\r\nThis is not fully reproducible, but almost all occurrences have been after some random pod causes OOMKiller to be triggered.\r\nWe've successfully quarantined the bug on a node, and can examine it if further information is needed.\r\n\r\n\r\n## Versions:\r\n```\r\n$ weave version\r\nweave 2.5.2\r\n$ docker version\r\n...\r\nServer: Docker Engine - Community\r\n Engine:\r\n  Version:          18.09.6\r\n  API version:      1.39 (minimum version 1.12)\r\n  Go version:       go1.10.8\r\n  Git commit:       481bc77\r\n  Built:            Sat May  4 01:59:36 2019\r\n  OS/Arch:          linux/amd64\r\n  Experimental:     false\r\n$ uname -a\r\nLinux c4-b2 4.4.0-119-generic #143-Ubuntu SMP Mon Apr 2 16:08:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n$ kubectl version\r\n...\r\nServer Version: version.Info{Major:\"1\", Minor:\"13\", GitVersion:\"v1.13.4\", GitCommit:\"c27b913fddd1a6c480c229191a087698aa92f0b1\", GitTreeState:\"clean\", BuildDate:\"2019-02-28T13:30:26Z\", GoVersion:\"go1.11.5\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n## Logs:\r\nLots of occurrences of the following lines:\r\n```\r\n...connection shutting down due to error: read tcp4...\r\n...connection deleted...\r\n...connection shutting down due to error during handshake: failed to receive remote protocol header...\r\n```\r\nbut these only appear during the OOM, after that it just goes back to normal logs (e.g. `Discovered remote MAC`)\r\n\r\n## Network:\r\n```\r\n$ ip route\r\n...\r\n10.32.0.0/12 dev weave  proto kernel  scope link  src 10.33.128.0\r\n$ ip -4 -o addr\r\n...\r\n6: weave    inet 10.33.128.0/12 brd 10.47.255.255 scope global weave\\       valid_lft forever preferred_lft forever\r\n$ /home/weave/weave --local status # inside the weave container\r\n        Version: 2.5.2 (up to date; next check at 2019/09/02 21:13:01)\r\n\r\n        Service: router\r\n       Protocol: weave 1..2\r\n           Name: 66:cf:c7:9d:f2:00(c4-b2)\r\n     Encryption: disabled\r\n  PeerDiscovery: enabled\r\n        Targets: 16\r\n    Connections: 16 (15 established, 1 failed)\r\n          Peers: 16 (with 240 established connections)\r\n TrustedSubnets: none\r\n\r\n        Service: ipam\r\n         Status: ready\r\n          Range: 10.32.0.0/12\r\n  DefaultSubnet: 10.32.0.0/12\r\n```"},"comment":{"url":"https://api.github.com/repos/weaveworks/weave/issues/comments/539879181","html_url":"https://github.com/weaveworks/weave/issues/3695#issuecomment-539879181","issue_url":"https://api.github.com/repos/weaveworks/weave/issues/3695","id":539879181,"node_id":"MDEyOklzc3VlQ29tbWVudDUzOTg3OTE4MQ==","user":{"login":"SeMeKh","id":1090449,"node_id":"MDQ6VXNlcjEwOTA0NDk=","avatar_url":"https://avatars0.githubusercontent.com/u/1090449?v=4","gravatar_id":"","url":"https://api.github.com/users/SeMeKh","html_url":"https://github.com/SeMeKh","followers_url":"https://api.github.com/users/SeMeKh/followers","following_url":"https://api.github.com/users/SeMeKh/following{/other_user}","gists_url":"https://api.github.com/users/SeMeKh/gists{/gist_id}","starred_url":"https://api.github.com/users/SeMeKh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SeMeKh/subscriptions","organizations_url":"https://api.github.com/users/SeMeKh/orgs","repos_url":"https://api.github.com/users/SeMeKh/repos","events_url":"https://api.github.com/users/SeMeKh/events{/privacy}","received_events_url":"https://api.github.com/users/SeMeKh/received_events","type":"User","site_admin":false},"created_at":"2019-10-09T07:38:56Z","updated_at":"2019-10-09T07:38:56Z","author_association":"NONE","body":"@bboreham Any ideas regarding this one? This is causing unexpected outages of our cluster."}}
{"actor":{"display_login":"SeMeKh"}}