{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/weave/issues/3384","repository_url":"https://api.github.com/repos/weaveworks/weave","labels_url":"https://api.github.com/repos/weaveworks/weave/issues/3384/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/weave/issues/3384/comments","events_url":"https://api.github.com/repos/weaveworks/weave/issues/3384/events","html_url":"https://github.com/weaveworks/weave/issues/3384","id":353458065,"node_id":"MDU6SXNzdWUzNTM0NTgwNjU=","number":3384,"title":"Weave not working correctly leads to containers stuck in ContainerCreating","user":{"login":"Raffo","id":683988,"node_id":"MDQ6VXNlcjY4Mzk4OA==","avatar_url":"https://avatars0.githubusercontent.com/u/683988?v=4","gravatar_id":"","url":"https://api.github.com/users/Raffo","html_url":"https://github.com/Raffo","followers_url":"https://api.github.com/users/Raffo/followers","following_url":"https://api.github.com/users/Raffo/following{/other_user}","gists_url":"https://api.github.com/users/Raffo/gists{/gist_id}","starred_url":"https://api.github.com/users/Raffo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Raffo/subscriptions","organizations_url":"https://api.github.com/users/Raffo/orgs","repos_url":"https://api.github.com/users/Raffo/repos","events_url":"https://api.github.com/users/Raffo/events{/privacy}","received_events_url":"https://api.github.com/users/Raffo/received_events","type":"User","site_admin":false},"labels":[{"id":627981433,"node_id":"MDU6TGFiZWw2Mjc5ODE0MzM=","url":"https://api.github.com/repos/weaveworks/weave/labels/state/need-more-info","name":"state/need-more-info","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2018-08-23T16:25:18Z","updated_at":"2018-09-04T18:12:32Z","closed_at":null,"author_association":"NONE","body":"## What you expected to happen?\r\n\r\nWeave should not have memory of previous removed nodes as this can cause ax exhaustion of IPs. \r\n\r\n## What happened?\r\n\r\nSome containers of the cluster were in status `ContainerCreating` and could never transion to a `Running` status. We could see by describing one of the pods that they were reporting `Failed create pod sandbox`. Here is a list of similar issues which could still be unrelated:\r\n\r\n- [https://github.com/kubernetes/kubeadm/issues/578](https://github.com/kubernetes/kubeadm/issues/578) (we don't use kubeadm, but it is related)\r\n- [https://github.com/kubernetes/kops/issues/3575](https://github.com/kubernetes/kops/issues/3575)\r\n- [https://github.com/weaveworks/weave/issues/3310](https://github.com/weaveworks/weave/issues/3310)\r\n- [https://github.com/weaveworks/weave/issues/3300](https://github.com/weaveworks/weave/issues/3300)\r\n\r\nIn our cluster we scale down the nodes every night to save money, by changing the size of the Autoscaling Group in AWS (it's a kops cluster).\r\n\r\nWe saw the following in weave containers:\r\n\r\n```\r\nfor i in $(kubectl get pods -n kube-system | grep weave | awk '{ print $1}'); do kubectl get pods $i -o wide -n kube-system; kubectl exec -n kube-system $i -c weave -- /home/weave/weave --local status connections; done\r\n\r\n<- 172.20.53.254:59238   established fastdp 9e:51:84:a9:2b:99(ip-172-20-53-254.eu-central-1.compute.internal) mtu=8912\r\n<- 172.20.125.51:52928   established fastdp aa:52:36:e7:8d:d3(ip-172-20-125-51.eu-central-1.compute.internal) mtu=8912\r\n<- 172.20.95.88:60391    established fastdp e2:a6:ae:06:8f:d1(ip-172-20-95-88.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.51.247:6783    failed      cannot connect to ourself, retry: never\r\n<- 172.20.53.254:33762   established fastdp 9e:51:84:a9:2b:99(ip-172-20-53-254.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.51.247:6783    established fastdp 6a:a4:ca:68:f4:02(ip-172-20-51-247.eu-central-1.compute.internal) mtu=8912\r\n<- 172.20.95.88:58856    established fastdp e2:a6:ae:06:8f:d1(ip-172-20-95-88.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.125.51:6783    failed      cannot connect to ourself, retry: never\r\n-> 172.20.51.247:6783    established fastdp 6a:a4:ca:68:f4:02(ip-172-20-51-247.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.125.51:6783    established fastdp aa:52:36:e7:8d:d3(ip-172-20-125-51.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.95.88:6783     established fastdp e2:a6:ae:06:8f:d1(ip-172-20-95-88.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.53.254:6783    failed      cannot connect to ourself, retry: never\r\n-> 172.20.125.51:6783    established fastdp aa:52:36:e7:8d:d3(ip-172-20-125-51.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.51.247:6783    established fastdp 6a:a4:ca:68:f4:02(ip-172-20-51-247.eu-central-1.compute.internal) mtu=8912\r\n<- 172.20.53.254:55665   established fastdp 9e:51:84:a9:2b:99(ip-172-20-53-254.eu-central-1.compute.internal) mtu=8912\r\n-> 172.20.95.88:6783     failed      cannot connect to ourself, retry: never\r\n```\r\n\r\nThis is **NOT** significant. It's fine that the nodes say that they can't connect to ourself or at least we see this error in the `status connection` command of the weave CLI even on a working cluster.\r\n\r\nWhat is more interesting is the output of `status ipam`:\r\n\r\n```\r\nkubectl exec -n kube-system weave-net-dcrj2 -c weave -- /home/weave/weave --local status ipam\r\n9e:51:84:a9:2b:99(ip-172-20-53-254.eu-central-1.compute.internal)        7 IPs (00.0% of total) (7 active)\r\nba:3e:73:6a:13:c7()                        256 IPs (00.0% of total) - unreachable!\r\na2:6a:83:e2:d2:7e()                         32 IPs (00.0% of total) - unreachable!\r\n32:a6:83:f6:c0:25()                       1024 IPs (00.0% of total) - unreachable!\r\n8e:19:b3:42:4a:ec()                       2048 IPs (00.1% of total) - unreachable!\r\nba:bb:8e:64:d8:07()                       4096 IPs (00.2% of total) - unreachable!\r\nda:f0:0a:b5:31:58()                     524288 IPs (25.0% of total) - unreachable!\r\nae:fc:8e:74:74:53()                       2048 IPs (00.1% of total) - unreachable!\r\n3e:a3:6c:2c:68:9c()                         16 IPs (00.0% of total) - unreachable!\r\n3e:bc:b5:42:15:66()                         32 IPs (00.0% of total) - unreachable!\r\n26:87:a6:1f:4c:82()                       8192 IPs (00.4% of total) - unreachable!\r\n82:cf:4e:23:3f:73()                       4096 IPs (00.2% of total) - unreachable!\r\nba:82:f4:d0:10:c5()                      32768 IPs (01.6% of total) - unreachable!\r\n62:06:bf:fa:c8:b2()                       4096 IPs (00.2% of total) - unreachable!\r\n5e:fd:cf:58:ce:01()                        256 IPs (00.0% of total) - unreachable!\r\n5a:f7:3b:61:39:61()                         32 IPs (00.0% of total) - unreachable!\r\n36:b5:90:80:65:88()                        512 IPs (00.0% of total) - unreachable!\r\n36:91:10:1e:29:de()                       1024 IPs (00.0% of total) - unreachable!\r\n62:0b:d4:f8:e1:51()                       4096 IPs (00.2% of total) - unreachable!\r\n9a:7c:fa:51:3b:a9()                        192 IPs (00.0% of total) - unreachable!\r\nda:c7:bd:46:98:c7()                       1024 IPs (00.0% of total) - unreachable!\r\ne6:cf:6c:3e:fb:b0()                       2048 IPs (00.1% of total) - unreachable!\r\n42:81:30:9e:df:0a()                        128 IPs (00.0% of total) - unreachable!\r\nfe:77:8f:46:67:f4()                       1024 IPs (00.0% of total) - unreachable!\r\n0e:85:43:e7:98:c2()                        512 IPs (00.0% of total) - unreachable!\r\n3a:83:86:eb:df:da()                        128 IPs (00.0% of total) - unreachable!\r\n16:41:a0:af:8c:3e()                        128 IPs (00.0% of total) - unreachable!\r\n3e:8c:be:be:a7:0c()                         16 IPs (00.0% of total) - unreachable!\r\nfa:88:5f:ea:c5:5f()                      65536 IPs (03.1% of total) - unreachable!\r\n9a:ba:ce:4d:60:bd()                       1024 IPs (00.0% of total) - unreachable!\r\nd6:ad:e3:03:aa:42()                         32 IPs (00.0% of total) - unreachable!\r\n56:db:68:38:9b:5b()                         32 IPs (00.0% of total) - unreachable!\r\n3a:0c:3c:e9:59:d8()                        128 IPs (00.0% of total) - unreachable!\r\nb6:76:96:73:bc:6b()                       2048 IPs (00.1% of total) - unreachable!\r\n1e:e8:8e:ad:fd:a9()                     262144 IPs (12.5% of total) - unreachable!\r\n8a:f0:9a:e1:c7:29()                         32 IPs (00.0% of total) - unreachable!\r\ne2:27:36:19:4e:c1()                      32768 IPs (01.6% of total) - unreachable!\r\n0e:bf:ce:ac:ea:dd()                        256 IPs (00.0% of total) - unreachable!\r\n8a:00:d6:3d:67:39()                        256 IPs (00.0% of total) - unreachable!\r\nae:03:57:54:c1:ec()                       2048 IPs (00.1% of total) - unreachable!\r\n1a:0d:d2:ff:88:3b()                      32768 IPs (01.6% of total) - unreachable!\r\n06:68:b6:87:48:75()                         64 IPs (00.0% of total) - unreachable!\r\n9e:f4:4f:b3:77:07()                       8192 IPs (00.4% of total) - unreachable!\r\n22:85:55:e9:07:e3()                         64 IPs (00.0% of total) - unreachable!\r\na6:cc:48:0b:42:8a()                        128 IPs (00.0% of total) - unreachable!\r\nfa:2e:36:62:23:d9()                       1024 IPs (00.0% of total) - unreachable!\r\nae:c8:70:e0:23:22()                      49152 IPs (02.3% of total) - unreachable!\r\nbe:66:9a:85:fa:df()                         16 IPs (00.0% of total) - unreachable!\r\n46:cb:ba:1c:b4:3a()                         16 IPs (00.0% of total) - unreachable!\r\nfa:00:d3:e8:a4:f1()                      32768 IPs (01.6% of total) - unreachable!\r\n8e:d7:cf:ff:97:69()                      16384 IPs (00.8% of total) - unreachable!\r\naa:52:36:e7:8d:d3(ip-172-20-125-51.eu-central-1.compute.internal)       19 IPs (00.0% of total)\r\nfe:05:22:50:04:0a()                       2048 IPs (00.1% of total) - unreachable!\r\n3e:91:da:4d:a9:ec()                     262144 IPs (12.5% of total) - unreachable!\r\n82:a3:c7:f9:6d:e9()                        128 IPs (00.0% of total) - unreachable!\r\n2e:8b:a6:cc:a7:19()                         32 IPs (00.0% of total) - unreachable!\r\n2e:f7:59:91:b2:11()                       4096 IPs (00.2% of total) - unreachable!\r\nc6:18:a6:97:97:4c()                      32768 IPs (01.6% of total) - unreachable!\r\n56:ab:99:e9:91:fd()                      16384 IPs (00.8% of total) - unreachable!\r\n7a:6d:41:17:b0:c3()                         20 IPs (00.0% of total) - unreachable!\r\nc2:7f:f3:07:bf:48()                       2048 IPs (00.1% of total) - unreachable!\r\n82:83:52:4f:34:f8()                     524288 IPs (25.0% of total) - unreachable!\r\n6a:a4:ca:68:f4:02(ip-172-20-51-247.eu-central-1.compute.internal)        2 IPs (00.0% of total)\r\n6a:09:6a:72:65:31()                         64 IPs (00.0% of total) - unreachable!\r\n3a:fe:7d:61:b6:12()                         32 IPs (00.0% of total) - unreachable!\r\n9e:93:78:0d:95:6f()                        512 IPs (00.0% of total) - unreachable!\r\na2:3e:3e:c8:40:34()                         16 IPs (00.0% of total) - unreachable!\r\n82:68:49:b6:38:28()                       4096 IPs (00.2% of total) - unreachable!\r\nc2:78:2d:27:b1:4d()                      16384 IPs (00.8% of total) - unreachable!\r\n76:5f:e2:06:fa:35()                     131072 IPs (06.2% of total) - unreachable!\r\n```\r\n\r\nThis seems to be telling us that most of the cluster is unreachable... which is making the CNI not work and containers can't start cause they can't get an IP address.\r\nWe verified that this was the case by reading the kubelet logs:\r\n\r\n\r\n```\r\nAug 23 07:46:29 ip-172-20-53-254 kubelet[7383]: E0823 07:46:29.891765    7383 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded\r\nAug 23 07:46:29 ip-172-20-53-254 kubelet[7383]: E0823 07:46:29.891815    7383 kuberuntime_sandbox.go:54] CreatePodSandbox for pod \"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded\r\nAug 23 07:46:29 ip-172-20-53-254 kubelet[7383]: E0823 07:46:29.891832    7383 kuberuntime_manager.go:647] createPodSandbox for pod \"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded\r\nAug 23 07:46:29 ip-172-20-53-254 kubelet[7383]: E0823 07:46:29.891888    7383 pod_workers.go:186] Error syncing pod f6fe3f93-a6a6-11e8-80a5-0205d2a81076 (\"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\"), skipping: failed to \"CreatePodSandbox\" for \"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\" with CreatePodSandboxError: \"CreatePodSandbox for pod \\\"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\\\" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded\"\r\nAug 23 07:46:30 ip-172-20-53-254 kubelet[7383]: I0823 07:46:30.730025    7383 kuberuntime_manager.go:416] Sandbox for pod \"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\" has no IP address.  Need to start a new one\r\nAug 23 07:46:31 ip-172-20-53-254 kubelet[7383]: I0823 07:46:31.436352    7383 kubelet.go:1896] SyncLoop (PLEG): \"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\", event: &pleg.PodLifecycleEvent{ID:\"f6fe3f93-a6a6-11e8-80a5-0205d2a81076\", Type:\"ContainerDied\", Data:\"da883b31b03187408bbee1b4642ba836932776977c200905fcb8e5f8cb9f4024\"}\r\nAug 23 07:46:31 ip-172-20-53-254 kubelet[7383]: W0823 07:46:31.436438    7383 pod_container_deletor.go:77] Container \"da883b31b03187408bbee1b4642ba836932776977c200905fcb8e5f8cb9f4024\" not found in pod's containers\r\nAug 23 07:46:31 ip-172-20-53-254 kubelet[7383]: I0823 07:46:31.436465    7383 kubelet.go:1896] SyncLoop (PLEG): \"nginx-7dc755b6f7-kc5g8_custom(f6fe3f93-a6a6-11e8-80a5-0205d2a81076)\", event: &pleg.PodLifecycleEvent{ID:\"f6fe3f93-a6a6-11e8-80a5-0205d2a81076\", Type:\"ContainerStarted\", Data:\"4deab2663ce209335c30401f003c0465401ef20604d32e2cfbd5ec6ab9b6b938\"}\r\nAug 23 07:47:05 ip-172-20-53-254 kubelet[7383]: I0823 07:47:05.109777    7383 server.go:796] GET /stats/summary/: (3.458746ms) 200 [[Go-http-client/1.1] 172.20.125.51:38646]\r\nAug 23 07:48:05 ip-172-20-53-254 kubelet[7383]: I0823 07:48:05.027382    7383 server.go:796] GET /stats/summary/: (3.582405ms) 200 [[Go-http-client/1.1] 172.20.125.51:38646]\r\nAug 23 07:48:26 ip-172-20-53-254 kubelet[7383]: I0823 07:48:26.863628    7383 container_manager_linux.go:425] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service\r\n```\r\n\r\nIn the logs above you can see `has no IP address.  Need to start a new one`.\r\n\r\nWe believe that this is due to the fact that we continuously shut down the nodes of our cluster in the night by simply scaling the ASG to 0 and back to the original size in the morning. It looks like that kops/weave do not do any automatic cleanup, probably cause they don't have a chance.\r\n\r\nFrom the `weave` documentation, it seems that we have to do something when the nodes exits, like mentioned in the [official documentation](https://www.weave.works/docs/net/latest/tasks/ipam/stop-remove-peers-ipam/). We still have to find a proper way to remove nodes from the Kubernetes cluster.\r\n\r\nWe did the reset by doing the following:\r\n\r\n- ssh into the EC2 instances (masters and workers) one by one and delete the file `/var/lib/weave/weave-netdata.db`. There is no need for a backup of that file\r\n- restart all the weave pods by deleting them, i.e.: `for i in $(kubectl get pods -n kube-system | awk '{print $1}' | grep weave); do kubectl delete pod -n kube-system $i; done`\r\n\r\nThis brought us back to a healthy state, that we could figure by running again the `status ipam` weave command:\r\n\r\n\r\n```\r\nk exec -it weave-net-47lhb -n kube-system -c weave /bin/sh\r\n/home/weave # ./weave --local status ipam\r\n9e:51:84:a9:2b:99(ip-172-20-53-254.eu-central-1.compute.internal)   524289 IPs (25.0% of total) (8 active)\r\n6a:a4:ca:68:f4:02(ip-172-20-51-247.eu-central-1.compute.internal)   786411 IPs (37.5% of total)\r\naa:52:36:e7:8d:d3(ip-172-20-125-51.eu-central-1.compute.internal)   524307 IPs (25.0% of total)\r\ne2:a6:ae:06:8f:d1(ip-172-20-95-88.eu-central-1.compute.internal)   262145 IPs (12.5% of total)\r\n```\r\n\r\n## How to reproduce it?\r\nNot sure, probably deleting lots of nodes from the cluster in a continuous way. \r\n\r\n\r\n## Anything else we need to know?\r\n<!-- Cloud provider? Hardware? How did you configure your cluster? Kubernetes YAML, KOPS, etc. -->\r\n\r\n## Versions:\r\n```\r\n$ weave version: 2.3.0\r\n$ docker version\r\n\r\nClient:\r\n Version:      17.03.2-ce\r\n API version:  1.27\r\n Go version:   go1.7.5\r\n Git commit:   f5ec1e2\r\n Built:        Tue Jun 27 03:35:14 2017\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      17.03.2-ce\r\n API version:  1.27 (minimum version 1.12)\r\n Go version:   go1.7.5\r\n Git commit:   f5ec1e2\r\n Built:        Tue Jun 27 03:35:14 2017\r\n OS/Arch:      linux/amd64\r\n Experimental: false\r\n\r\n$ uname -a\r\nLinux ip-172-20-95-88 4.4.0-1054-aws #63-Ubuntu SMP Wed Mar 28 19:42:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n$ kubectl version\r\nServer Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.6\", GitCommit:\"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\", GitTreeState:\"clean\", BuildDate:\"2018-03-21T15:13:31Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n## Logs:\r\n\r\nI don't have other logs to paste for the moment. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/weaveworks/weave/issues/comments/418466130","html_url":"https://github.com/weaveworks/weave/issues/3384#issuecomment-418466130","issue_url":"https://api.github.com/repos/weaveworks/weave/issues/3384","id":418466130,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODQ2NjEzMA==","user":{"login":"dcowden","id":1297923,"node_id":"MDQ6VXNlcjEyOTc5MjM=","avatar_url":"https://avatars0.githubusercontent.com/u/1297923?v=4","gravatar_id":"","url":"https://api.github.com/users/dcowden","html_url":"https://github.com/dcowden","followers_url":"https://api.github.com/users/dcowden/followers","following_url":"https://api.github.com/users/dcowden/following{/other_user}","gists_url":"https://api.github.com/users/dcowden/gists{/gist_id}","starred_url":"https://api.github.com/users/dcowden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcowden/subscriptions","organizations_url":"https://api.github.com/users/dcowden/orgs","repos_url":"https://api.github.com/users/dcowden/repos","events_url":"https://api.github.com/users/dcowden/events{/privacy}","received_events_url":"https://api.github.com/users/dcowden/received_events","type":"User","site_admin":false},"created_at":"2018-09-04T18:12:32Z","updated_at":"2018-09-04T18:12:32Z","author_association":"NONE","body":"@bboreham given that we cycle through our cluster frequently, and ran into this issue, we would like to implement a workaround until the issue is resolved.  My understanding from reading most of the issues is that part of the difficulty in applying a fix is that it is hard to know when a node has gone permanently and when it is temporary.\r\n\r\nIn our case, we control the process of terminating nodes, so we know when it is.  Our script drains and then terminates nodes in a predictable order.\r\n\r\nIf we could do something at the time we terminate a node, what would the 'right thing' to do be?\r\n\r\nwould it work to simply run` 'weave reset'` on each node after we finish restarting them all? \r\n\r\nOr would it be better to randomly select a node and run `'weave rmpeer <hostname we just removed>` on one of them?\r\n\r\n\r\nOne of our annoying details is that we can no longer run kubectl exec on the weave pods-- they are using hostNetwork true, so we can't exec on them remotely.\r\n"}}
{"actor":{"display_login":"dcowden"}}