{"action":"closed","issue":{"url":"https://api.github.com/repos/projectcalico/calico/issues/1682","repository_url":"https://api.github.com/repos/projectcalico/calico","labels_url":"https://api.github.com/repos/projectcalico/calico/issues/1682/labels{/name}","comments_url":"https://api.github.com/repos/projectcalico/calico/issues/1682/comments","events_url":"https://api.github.com/repos/projectcalico/calico/issues/1682/events","html_url":"https://github.com/projectcalico/calico/issues/1682","id":295734276,"number":1682,"title":" The number of pod ips on nodes is not consistent with the number in etcd","user":{"login":"mdianjun","id":12122541,"avatar_url":"https://avatars2.githubusercontent.com/u/12122541?v=4","gravatar_id":"","url":"https://api.github.com/users/mdianjun","html_url":"https://github.com/mdianjun","followers_url":"https://api.github.com/users/mdianjun/followers","following_url":"https://api.github.com/users/mdianjun/following{/other_user}","gists_url":"https://api.github.com/users/mdianjun/gists{/gist_id}","starred_url":"https://api.github.com/users/mdianjun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdianjun/subscriptions","organizations_url":"https://api.github.com/users/mdianjun/orgs","repos_url":"https://api.github.com/users/mdianjun/repos","events_url":"https://api.github.com/users/mdianjun/events{/privacy}","received_events_url":"https://api.github.com/users/mdianjun/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-02-09T02:26:33Z","updated_at":"2018-02-15T18:51:28Z","closed_at":"2018-02-15T18:51:28Z","author_association":"NONE","body":"The node 10.196.104.195 has 2 pod ips: 10.163.72.209 and 10.163.72.215 as follows:\r\n```\r\n# kubectl get --all-namespaces pods -owide |grep 104-195\r\ndefault       centos-daemon-h9q2q                                                   1/1       Running             0          6d        10.163.72.209    kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nkube-system   calico-etcd-kc-bcc-pod23-10-196-104-195.hadoop.jd.local               1/1       Running             0          16h       10.196.104.195   kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nkube-system   calico-node-pod23-cktbm                                               2/2       Running             0          13h       10.196.104.195   kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nkube-system   event-etcd-kc-bcc-pod23-10-196-104-195.hadoop.jd.local                1/1       Running             0          16h       10.196.104.195   kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nkube-system   kube-dns-d745dc87b-24fgc                                              3/3       Running             0          2d        10.163.72.215    kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nkube-system   kube-external-dns-qlgkx                                               3/3       Running             0          27d       10.196.104.195   kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nkube-system   kube-nginx-kc-bcc-pod23-10-196-104-195.hadoop.jd.local                1/1       Running             0          2d        10.196.104.195   kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nkube-system   kube-proxy-9tn5c                                                      1/1       Running             0          28d       10.196.104.195   kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\nmonitor       node-exporter-hpg9f                                                   1/1       Running             0          27d       10.196.104.195   kc-bcc-pod23-10-196-104-195.hadoop.jd.local\r\n# ip route  // view routes on 10.196.104.195 \r\ndefault via 10.196.104.193 dev eth0 \r\nblackhole 10.163.72.208/28  proto bird \r\n10.163.72.209 dev cali86083622e28  scope link \r\n10.163.72.215 dev cali6556d7588e1  scope link \r\n10.196.104.192/27 dev eth0  proto kernel  scope link  src 10.196.104.195 \r\n169.254.0.0/16 dev eth0  scope link  metric 1002 \r\n169.254.0.0/16 dev enp2s0f1  scope link  metric 1003 \r\n192.168.0.0/16 dev docker0  proto kernel  scope link  src 192.168.0.1\r\n```\r\nThe block in etcd has allocated 3 ips to that node:\r\n```\r\n# etcdctl --endpoints http://10.196.104.195:6666 get --prefix --keys-only=false /calico/ipam/v2/assignment/ipv4/block/10.163.72.208-28\r\n/calico/ipam/v2/assignment/ipv4/block/10.163.72.208-28\r\n{\"cidr\":\"10.163.72.208/28\",\"affinity\":\"host:kc-bcc-pod23-10-196-104-195.hadoop.jd.local\",\"strictAffinity\":true,\"allocations\":[null,1,null,null,null,null,null,2,null,0,null,null,null,null,null,null],\"unallocated\":[10,11,12,14,13,3,15,0,2,6,4,8,5],\"attributes\":[{\"handle_id\":\"k8s-pod-network.11b1d669e2303016a4221dbd816a076e623ddb5ceffa9a1a582e3995bce0ab15\",\"secondary\":null},{\"handle_id\":\"k8s-pod-network.66846bf0392aa038153e6a0210c84963302eb17ecfb09f3f1de89fba4aaea827\",\"secondary\":null},{\"handle_id\":\"k8s-pod-network.53be02be40d51f5c61f34fecd17f04675a32534c1d405a4610654e024c386530\",\"secondary\":null}]}\r\n``` \r\n\r\nIn my cluster with 306 nodes, there are 6 nodes which the above situation appears on. It seems that calico-cni didn't release ip successfully. \r\n"}}
{"actor":{"display_login":"caseydavenport"}}