{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57280","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57280/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57280/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57280/events","html_url":"https://github.com/kubernetes/kubernetes/issues/57280","id":282658961,"number":57280,"title":"Kubernetes-cni issue with 1.9.0 - no ip address available in range","user":{"login":"ieugen","id":1050920,"avatar_url":"https://avatars0.githubusercontent.com/u/1050920?v=4","gravatar_id":"","url":"https://api.github.com/users/ieugen","html_url":"https://github.com/ieugen","followers_url":"https://api.github.com/users/ieugen/followers","following_url":"https://api.github.com/users/ieugen/following{/other_user}","gists_url":"https://api.github.com/users/ieugen/gists{/gist_id}","starred_url":"https://api.github.com/users/ieugen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ieugen/subscriptions","organizations_url":"https://api.github.com/users/ieugen/orgs","repos_url":"https://api.github.com/users/ieugen/repos","events_url":"https://api.github.com/users/ieugen/events{/privacy}","received_events_url":"https://api.github.com/users/ieugen/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":116712108,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false}],"state":"closed","locked":false,"assignee":{"login":"cmluciano","id":4790487,"avatar_url":"https://avatars1.githubusercontent.com/u/4790487?v=4","gravatar_id":"","url":"https://api.github.com/users/cmluciano","html_url":"https://github.com/cmluciano","followers_url":"https://api.github.com/users/cmluciano/followers","following_url":"https://api.github.com/users/cmluciano/following{/other_user}","gists_url":"https://api.github.com/users/cmluciano/gists{/gist_id}","starred_url":"https://api.github.com/users/cmluciano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmluciano/subscriptions","organizations_url":"https://api.github.com/users/cmluciano/orgs","repos_url":"https://api.github.com/users/cmluciano/repos","events_url":"https://api.github.com/users/cmluciano/events{/privacy}","received_events_url":"https://api.github.com/users/cmluciano/received_events","type":"User","site_admin":false},"assignees":[{"login":"cmluciano","id":4790487,"avatar_url":"https://avatars1.githubusercontent.com/u/4790487?v=4","gravatar_id":"","url":"https://api.github.com/users/cmluciano","html_url":"https://github.com/cmluciano","followers_url":"https://api.github.com/users/cmluciano/followers","following_url":"https://api.github.com/users/cmluciano/following{/other_user}","gists_url":"https://api.github.com/users/cmluciano/gists{/gist_id}","starred_url":"https://api.github.com/users/cmluciano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmluciano/subscriptions","organizations_url":"https://api.github.com/users/cmluciano/orgs","repos_url":"https://api.github.com/users/cmluciano/repos","events_url":"https://api.github.com/users/cmluciano/events{/privacy}","received_events_url":"https://api.github.com/users/cmluciano/received_events","type":"User","site_admin":false}],"milestone":null,"comments":21,"created_at":"2017-12-16T23:55:37Z","updated_at":"2018-02-07T05:56:35Z","closed_at":"2018-01-23T20:41:29Z","author_association":"NONE","body":"**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n\r\n/kind bug\r\n\r\n**What happened**:\r\nI've upgraded my cluster from 1.8.5 -> 1.9.0. I upgraded the packages on my system and then used the kubeadm upgrade instructions. The upgrade went well. I changed docker logging config and restarted docker. Kube-dns did not start and I had two of them. Also my deployments did not start as well.\r\n\r\n**What you expected to happen**:\r\nI expected my cluster to work normally as expected. \r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n1. Create new cluster with 1.8.5.\r\n2. Update packages to 1.9.0\r\n3. kubeadm upgrade plan\r\n4. kubeadm upgrade apply 1.9.0\r\n5. weep.\r\n\r\n**Anything else we need to know?**:\r\n\r\nThe main issue is that pods don't find an IP address: \r\n````\r\n E1216 23:50:16.116098   28152 pod_workers.go:186] Error syncing pod 6f5b9673-e2b5-11e7-a0f5-001e67d35991 (\"kube-dns-6f4fd4bdf-xrj4w_kube-system(6f5b9673-e2b5-11e7-a0f5-001e67d35991)\"), skipping: failed to \"CreatePodSandbox\" for \"kube-dns-6f4fd4bdf-xrj4w_kube-system(6f5b9673-e2b5-11e7-a0f5-001e67d35991)\" with CreatePodSandboxError: \"CreatePodSandbox for pod \\\"kube-dns-6f4fd4bdf-xrj4w_kube-system(6f5b9673-e2b5-11e7-a0f5-001e67d35991)\\\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \\\"kube-dns-6f4fd4bdf-xrj4w_kube-system\\\" network: failed to allocate for range 0: no IP addresses available in range set: 10.244.0.1-10.244.0.254\"\r\n````\r\nNow, I have checked to see what happens in `/var/lib/cni/networks` and I have found that all the addresses are filled up:\r\n````\r\n/var/lib/cni/networks# ls cbr0/\r\n10.244.0.10   10.244.0.123  10.244.0.147  10.244.0.170\t10.244.0.194  10.244.0.217  10.244.0.240  10.244.0.35  10.244.0.59  10.244.0.82\r\n10.244.0.100  10.244.0.124  10.244.0.148  10.244.0.171\t10.244.0.195  10.244.0.218  10.244.0.241  10.244.0.36  10.244.0.6   10.244.0.83\r\n10.244.0.101  10.244.0.125  10.244.0.149  10.244.0.172\t10.244.0.196  10.244.0.219  10.244.0.242  10.244.0.37  10.244.0.60  10.244.0.84\r\n10.244.0.102  10.244.0.126  10.244.0.15   10.244.0.173\t10.244.0.197  10.244.0.22   10.244.0.243  10.244.0.38  10.244.0.61  10.244.0.85\r\n10.244.0.103  10.244.0.127  10.244.0.150  10.244.0.174\t10.244.0.198  10.244.0.220  10.244.0.244  10.244.0.39  10.244.0.62  10.244.0.86\r\n10.244.0.104  10.244.0.128  10.244.0.151  10.244.0.175\t10.244.0.199  10.244.0.221  10.244.0.245  10.244.0.4   10.244.0.63  10.244.0.87\r\n10.244.0.105  10.244.0.129  10.244.0.152  10.244.0.176\t10.244.0.2    10.244.0.222  10.244.0.246  10.244.0.40  10.244.0.64  10.244.0.88\r\n10.244.0.106  10.244.0.13   10.244.0.153  10.244.0.177\t10.244.0.20   10.244.0.223  10.244.0.247  10.244.0.41  10.244.0.65  10.244.0.89\r\n10.244.0.107  10.244.0.130  10.244.0.154  10.244.0.178\t10.244.0.200  10.244.0.224  10.244.0.248  10.244.0.42  10.244.0.66  10.244.0.9\r\n10.244.0.108  10.244.0.131  10.244.0.155  10.244.0.179\t10.244.0.201  10.244.0.225  10.244.0.249  10.244.0.43  10.244.0.67  10.244.0.90\r\n10.244.0.109  10.244.0.132  10.244.0.156  10.244.0.18\t10.244.0.202  10.244.0.226  10.244.0.25   10.244.0.44  10.244.0.68  10.244.0.91\r\n10.244.0.11   10.244.0.133  10.244.0.157  10.244.0.180\t10.244.0.203  10.244.0.227  10.244.0.250  10.244.0.45  10.244.0.69  10.244.0.92\r\n10.244.0.110  10.244.0.134  10.244.0.158  10.244.0.181\t10.244.0.204  10.244.0.228  10.244.0.251  10.244.0.46  10.244.0.7   10.244.0.93\r\n10.244.0.111  10.244.0.135  10.244.0.159  10.244.0.182\t10.244.0.205  10.244.0.229  10.244.0.252  10.244.0.47  10.244.0.70  10.244.0.94\r\n10.244.0.112  10.244.0.136  10.244.0.16   10.244.0.183\t10.244.0.206  10.244.0.23   10.244.0.253  10.244.0.48  10.244.0.71  10.244.0.95\r\n10.244.0.113  10.244.0.137  10.244.0.160  10.244.0.184\t10.244.0.207  10.244.0.230  10.244.0.254  10.244.0.49  10.244.0.72  10.244.0.96\r\n10.244.0.114  10.244.0.138  10.244.0.161  10.244.0.185\t10.244.0.208  10.244.0.231  10.244.0.26   10.244.0.5   10.244.0.73  10.244.0.97\r\n10.244.0.115  10.244.0.139  10.244.0.162  10.244.0.186\t10.244.0.209  10.244.0.232  10.244.0.27   10.244.0.50  10.244.0.74  10.244.0.98\r\n10.244.0.116  10.244.0.14   10.244.0.163  10.244.0.187\t10.244.0.21   10.244.0.233  10.244.0.28   10.244.0.51  10.244.0.75  10.244.0.99\r\n10.244.0.117  10.244.0.140  10.244.0.164  10.244.0.188\t10.244.0.210  10.244.0.234  10.244.0.29   10.244.0.52  10.244.0.76  last_reserved_ip.0\r\n10.244.0.118  10.244.0.141  10.244.0.165  10.244.0.189\t10.244.0.211  10.244.0.235  10.244.0.3\t  10.244.0.53  10.244.0.77\r\n10.244.0.119  10.244.0.142  10.244.0.166  10.244.0.19\t10.244.0.212  10.244.0.236  10.244.0.30   10.244.0.54  10.244.0.78\r\n10.244.0.12   10.244.0.143  10.244.0.167  10.244.0.190\t10.244.0.213  10.244.0.237  10.244.0.31   10.244.0.55  10.244.0.79\r\n10.244.0.120  10.244.0.144  10.244.0.168  10.244.0.191\t10.244.0.214  10.244.0.238  10.244.0.32   10.244.0.56  10.244.0.8\r\n10.244.0.121  10.244.0.145  10.244.0.169  10.244.0.192\t10.244.0.215  10.244.0.239  10.244.0.33   10.244.0.57  10.244.0.80\r\n10.244.0.122  10.244.0.146  10.244.0.17   10.244.0.193\t10.244.0.216  10.244.0.24   10.244.0.34   10.244.0.58  10.244.0.81\r\n````\r\n and flannel creates a lot of files without stopping:\r\n````\r\n/var/lib/cni/flannel#  ls | wc ; date \r\n   1207    1207   78455\r\nSat Dec 16 23:53:25 UTC 2017\r\nroot@staging:/var/lib/cni/flannel#  ls | wc ; date \r\n   1212    1212   78780\r\nSat Dec 16 23:53:27 UTC 2017\r\nroot@staging:/var/lib/cni/flannel#  ls | wc ; date \r\n   1214    1214   78910\r\nSat Dec 16 23:53:28 UTC 2017\r\n````\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n````\r\nkubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.0\", GitCommit:\"925c127ec6b946659ad0fd596fa959be43f0cc05\", GitTreeState:\"clean\", BuildDate:\"2017-12-15T21:07:38Z\", GoVersion:\"go1.9.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.0\", GitCommit:\"925c127ec6b946659ad0fd596fa959be43f0cc05\", GitTreeState:\"clean\", BuildDate:\"2017-12-15T20:55:30Z\", GoVersion:\"go1.9.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n````\r\n- Cloud provider or hardware configuration: bare metal, single node\r\n- OS (e.g. from /etc/os-release): \r\n````\r\nPRETTY_NAME=\"Debian GNU/Linux 9 (stretch)\"\r\nNAME=\"Debian GNU/Linux\"\r\nVERSION_ID=\"9\"\r\nVERSION=\"9 (stretch)\"\r\nID=debian\r\n````\r\n- Kernel (e.g. `uname -a`):  4.9.0-4-amd64 #1 SMP Debian 4.9.65-3 (2017-12-03) x86_64 GNU/Linux\r\n- Install tools: kubeadm\r\n- Others: flannel 0.9.1 \r\n\r\nFor some reason the system ends up with 2 `deploy/kube-dns` and \r\n````\r\n kubectl -n kube-system get all \r\nNAME                 DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE\r\nds/kube-flannel-ds   1         1         1         1            1           beta.kubernetes.io/arch=amd64   33m\r\nds/kube-proxy        1         1         1         1            1           <none>                          38m\r\n\r\nNAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE\r\ndeploy/kube-dns               1         2         1            1           38m\r\ndeploy/kubernetes-dashboard   1         1         1            1           30m\r\n\r\nNAME                                DESIRED   CURRENT   READY     AGE\r\nrs/kube-dns-545bc4bfd4              1         1         1         38m\r\nrs/kube-dns-6f4fd4bdf               1         1         0         4m\r\nrs/kubernetes-dashboard-79ddfdc44   1         1         1         30m\r\n\r\nNAME                 DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE\r\nds/kube-flannel-ds   1         1         1         1            1           beta.kubernetes.io/arch=amd64   33m\r\nds/kube-proxy        1         1         1         1            1           <none>                          38m\r\n\r\nNAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE\r\ndeploy/kube-dns               1         2         1            1           38m\r\ndeploy/kubernetes-dashboard   1         1         1            1           30m\r\n\r\nNAME                                DESIRED   CURRENT   READY     AGE\r\nrs/kube-dns-545bc4bfd4              1         1         1         38m\r\nrs/kube-dns-6f4fd4bdf               1         1         0         4m\r\nrs/kubernetes-dashboard-79ddfdc44   1         1         1         30m\r\n\r\nNAME                                           READY     STATUS              RESTARTS   AGE\r\npo/etcd-staging.gr8pi.net                      1/1       Running             0          3m\r\npo/kube-apiserver-staging.gr8pi.net            1/1       Running             0          4m\r\npo/kube-controller-manager-staging.gr8pi.net   1/1       Running             0          4m\r\npo/kube-dns-545bc4bfd4-xs7zw                   3/3       Running             6          38m\r\npo/kube-dns-6f4fd4bdf-wvmgp                    0/3       ContainerCreating   0          2m\r\npo/kube-flannel-ds-8nb76                       1/1       Running             3          33m\r\npo/kube-proxy-fvtzr                            1/1       Running             0          4m\r\npo/kube-scheduler-staging.gr8pi.net            1/1       Running             0          4m\r\npo/kubernetes-dashboard-79ddfdc44-p6sfv        1/1       Running             4          30m\r\n\r\nNAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE\r\nsvc/kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   38m\r\nsvc/kubernetes-dashboard   NodePort    10.103.193.125   <none>        80:30940/TCP    30m\r\n````\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/363665818","html_url":"https://github.com/kubernetes/kubernetes/issues/57280#issuecomment-363665818","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57280","id":363665818,"user":{"login":"brentpearson","id":3093641,"avatar_url":"https://avatars2.githubusercontent.com/u/3093641?v=4","gravatar_id":"","url":"https://api.github.com/users/brentpearson","html_url":"https://github.com/brentpearson","followers_url":"https://api.github.com/users/brentpearson/followers","following_url":"https://api.github.com/users/brentpearson/following{/other_user}","gists_url":"https://api.github.com/users/brentpearson/gists{/gist_id}","starred_url":"https://api.github.com/users/brentpearson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brentpearson/subscriptions","organizations_url":"https://api.github.com/users/brentpearson/orgs","repos_url":"https://api.github.com/users/brentpearson/repos","events_url":"https://api.github.com/users/brentpearson/events{/privacy}","received_events_url":"https://api.github.com/users/brentpearson/received_events","type":"User","site_admin":false},"created_at":"2018-02-07T05:56:35Z","updated_at":"2018-02-07T05:56:35Z","author_association":"NONE","body":"\"I recommend you configure your host-local ipam plugin to use a dataDir in /run, instead of the default in /var/lib. That will mean that IPAM is cleared on every reboot.\"\r\n\r\nCould someone please explain how to do this?"}}
{"actor":{"display_login":"brentpearson"}}