{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21401","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21401/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21401/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21401/events","html_url":"https://github.com/kubernetes/kubernetes/issues/21401","id":134353804,"number":21401,"title":"GKE: Kubelet not configuring CBR0 on 4 of 1000 nodes","user":{"login":"alex-mohr","id":8729097,"avatar_url":"https://avatars.githubusercontent.com/u/8729097?v=3","gravatar_id":"","url":"https://api.github.com/users/alex-mohr","html_url":"https://github.com/alex-mohr","followers_url":"https://api.github.com/users/alex-mohr/followers","following_url":"https://api.github.com/users/alex-mohr/following{/other_user}","gists_url":"https://api.github.com/users/alex-mohr/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-mohr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-mohr/subscriptions","organizations_url":"https://api.github.com/users/alex-mohr/orgs","repos_url":"https://api.github.com/users/alex-mohr/repos","events_url":"https://api.github.com/users/alex-mohr/events{/privacy}","received_events_url":"https://api.github.com/users/alex-mohr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/gke","name":"team/gke","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/15","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.2","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/15/labels","id":1278499,"number":15,"title":"v1.2","description":"https://github.com/kubernetes/kubernetes/wiki/Release-1.2","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":165,"closed_issues":192,"state":"open","created_at":"2015-08-29T03:54:16Z","updated_at":"2016-02-17T17:59:19Z","due_on":"2016-03-07T08:00:00Z","closed_at":null},"comments":3,"created_at":"2016-02-17T17:59:19Z","updated_at":"2016-02-17T18:08:49Z","closed_at":null,"body":"From head @ Feb 16 ~9am, a 1000 node cluster on GKE had 4 nodes that came up as persistently flapping healthy/unhealthy.  They cycle through a loop where apparently kubelet restarts itself because docker is not running.  Logging details follow with instructions on how to access the node at the end.\r\n```\r\n% gcloud container clusters get-credentials mohr-test --zone=us-central1-c --project=mohr-instance-manager1\r\n% kubectl describe node gke-mohr-test-a83d8da2-node-yv6i\r\n...\r\n 15m           15m             1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      Starting                Starting kubelet.\r\n  15m           15m             1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      NodeHasSufficientDisk   Node gke-mohr-test-a83d8da2-node-yv6i status is now: NodeHasSufficientDisk\r\n  15m           15m             1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      NodeReady               Node gke-mohr-test-a83d8da2-node-yv6i status is now: NodeReady\r\n  10m           10m             1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      NodeNotReady            Node gke-mohr-test-a83d8da2-node-yv6i status is now: NodeNotReady\r\n  5m            5m              1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      Starting                Starting kubelet.\r\n  5m            5m              1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      NodeHasSufficientDisk   Node gke-mohr-test-a83d8da2-node-yv6i status is now: NodeHasSufficientDisk\r\n  4m            4m              1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      NodeReady               Node gke-mohr-test-a83d8da2-node-yv6i status is now: NodeReady\r\n  51s           51s             1       {kubelet gke-mohr-test-a83d8da2-node-yv6i}                      NodeNotReady            Node gke-mohr-test-a83d8da2-node-yv6i status is now: NodeNotReady\r\n```\r\nI don't understand kubelet and docker setup well enough to debug, so here are logs.  Most telling might be docker.log, full of repeated entries (Note that kube-env has CLUSTER_IP_RANGE: 10.132.0.0/14):\r\n```\r\ntime=\"2016-02-17T17:50:27.304870401Z\" level=warning msg=\"Could not get list of networks during endpoint cleanup: could not find endpoint count key docker/network/v1.0/endpoint_count/3bda6e7914e3c4986a9c7557aef98064035952e8376d1bb1e692dd45fab91e77/ for network bridge while listing: Key not found in store\" \r\ntime=\"2016-02-17T17:50:27.305104444Z\" level=error msg=\"could not find endpoint count key docker/network/v1.0/endpoint_count/3bda6e7914e3c4986a9c7557aef98064035952e8376d1bb1e692dd45fab91e77/ for network bridge while listing: Key not found in store\" \r\ntime=\"2016-02-17T17:50:27.309913613Z\" level=fatal msg=\"Error starting daemon: Error initializing network controller: Error creating default \\\"bridge\\\" network: failed to allocate gateway (10.132.184.1): No available addresses on this pool\" \r\n```\r\nFrom gke-mohr-test-a83d8da2-node-yv6's kubelet.log, there seems to be some strangeness with configuring cbr0:\r\n```\r\ncmdline: /usr/local/bin/kubelet --api-servers=https://104.197.237.207 --enable-debugging-handlers=true --cloud-provider=gce --config=/etc/kubernetes/manifests --allow-privileged=True --v=2 --cluster-dns=10.135.240.10 --cluster-domain=cluster.local --configure-cbr0=true --cgroup-root=/ --system-cgroups=/system --configure-hairpin-mode=true\r\n```\r\n```\r\nW0217 00:23:00.613889    1482 kubelet.go:2687] ConfigureCBR0 requested, but PodCIDR not set. Will not configure CBR0 right now\r\nI0217 00:23:01.066330    1482 kubelet.go:1018] Adding node label from cloud provider: beta.kubernetes.io/instance-type=n1-standard-1\r\nI0217 00:23:01.066348    1482 kubelet.go:1029] Adding node label from cloud provider: failure-domain.alpha.kubernetes.io/zone=us-central1-c\r\nI0217 00:23:01.066353    1482 kubelet.go:1033] Adding node label from cloud provider: failure-domain.alpha.kubernetes.io/region=us-central1\r\nE0217 00:23:01.067745    1482 kubelet.go:2843] Error getting image list: cannot connect to Docker endpoint\r\nI0217 00:23:01.067778    1482 kubelet.go:2661] Recording NodeHasSufficientDisk event message for node gke-mohr-test-a83d8da2-node-yv6i\r\nI0217 00:23:01.067894    1482 kubelet.go:1066] Attempting to register node gke-mohr-test-a83d8da2-node-yv6i\r\nI0217 00:23:01.074493    1482 kubelet.go:1082] Node gke-mohr-test-a83d8da2-node-yv6i was previously registered\r\nI0217 00:23:01.076974    1482 kubelet.go:3480] Setting Pod CIDR:  -> 10.132.184.0/24\r\nE0217 00:23:01.078254    1482 kubelet.go:2843] Error getting image list: cannot connect to Docker endpoint\r\nE0217 00:23:01.614173    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nE0217 00:23:02.614561    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nE0217 00:23:03.615012    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nE0217 00:23:04.615405    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nI0217 00:23:05.614176    1482 kubelet.go:2287] skipping pod synchronization - [ConfigureCBR0 requested, but PodCIDR not set. Will not configure CBR0 right now]\r\nE0217 00:23:05.614382    1482 kubelet.go:2617] Container runtime sanity check failed: docker: failed to get docker version: cannot connect to Docker endpoint\r\nE0217 00:23:05.615717    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nE0217 00:23:06.616077    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nE0217 00:23:07.616453    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nE0217 00:23:08.616805    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nE0217 00:23:09.617125    1482 generic.go:164] GenericPLEG: Unable to retrieve pods: cannot connect to Docker endpoint\r\nI0217 00:23:10.616219    1482 kubelet.go:2287] skipping pod synchronization - [ConfigureCBR0 requested, but PodCIDR not set. Will not configure CBR0 right now]\r\n```\r\nList of nodes\r\n```\r\n% kubectl get nodes | grep \"NotReady\" \r\ngke-mohr-test-a83d8da2-node-afiv   beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.alpha.kubernetes.io/region=us-central1,failure-domain.alpha.kubernetes.io/zone=us-central1-c,kubernetes.io/hostname=gke-mohr-test-a83d8da2-node-afiv   NotReady   22h\r\ngke-mohr-test-a83d8da2-node-qu7s   beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.alpha.kubernetes.io/region=us-central1,failure-domain.alpha.kubernetes.io/zone=us-central1-c,kubernetes.io/hostname=gke-mohr-test-a83d8da2-node-qu7s   NotReady   22h\r\ngke-mohr-test-a83d8da2-node-xi4t   beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.alpha.kubernetes.io/region=us-central1,failure-domain.alpha.kubernetes.io/zone=us-central1-c,kubernetes.io/hostname=gke-mohr-test-a83d8da2-node-xi4t   NotReady   22h\r\ngke-mohr-test-a83d8da2-node-yv6i   beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.alpha.kubernetes.io/region=us-central1,failure-domain.alpha.kubernetes.io/zone=us-central1-c,kubernetes.io/hostname=gke-mohr-test-a83d8da2-node-yv6i   NotReady   22h\r\n```\r\nI've added @dchen1107 and @thockin and most of the Google team to my project, so you can access the node via the following -- and let me know if you need more.  I'll assign to Dawn to triage, but apologies if someone else should have been assigned; let me know who that would have been and I'll do better next time.\r\n```\r\n% gcloud compute --project \"mohr-instance-manager1\" ssh --zone \"us-central1-c\" \"gke-mohr-test-a83d8da2-node-yv6i\"\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/185329098","html_url":"https://github.com/kubernetes/kubernetes/issues/21401#issuecomment-185329098","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21401","id":185329098,"user":{"login":"cjcullen","id":7154008,"avatar_url":"https://avatars.githubusercontent.com/u/7154008?v=3","gravatar_id":"","url":"https://api.github.com/users/cjcullen","html_url":"https://github.com/cjcullen","followers_url":"https://api.github.com/users/cjcullen/followers","following_url":"https://api.github.com/users/cjcullen/following{/other_user}","gists_url":"https://api.github.com/users/cjcullen/gists{/gist_id}","starred_url":"https://api.github.com/users/cjcullen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjcullen/subscriptions","organizations_url":"https://api.github.com/users/cjcullen/orgs","repos_url":"https://api.github.com/users/cjcullen/repos","events_url":"https://api.github.com/users/cjcullen/events{/privacy}","received_events_url":"https://api.github.com/users/cjcullen/received_events","type":"User","site_admin":false},"created_at":"2016-02-17T18:08:48Z","updated_at":"2016-02-17T18:08:48Z","body":"Possibly related? https://github.com/docker/docker/issues/18113"}}