{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21367","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21367/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21367/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21367/events","html_url":"https://github.com/kubernetes/kubernetes/issues/21367","id":134153730,"number":21367,"title":"e2e flake: failed to observe pod deletion - should be submitted and removed [Conformance]","user":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars.githubusercontent.com/u/7085343?v=3","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/flake","name":"kind/flake","color":"f7c6c7"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"mikedanese","id":1787169,"avatar_url":"https://avatars.githubusercontent.com/u/1787169?v=3","gravatar_id":"","url":"https://api.github.com/users/mikedanese","html_url":"https://github.com/mikedanese","followers_url":"https://api.github.com/users/mikedanese/followers","following_url":"https://api.github.com/users/mikedanese/following{/other_user}","gists_url":"https://api.github.com/users/mikedanese/gists{/gist_id}","starred_url":"https://api.github.com/users/mikedanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikedanese/subscriptions","organizations_url":"https://api.github.com/users/mikedanese/orgs","repos_url":"https://api.github.com/users/mikedanese/repos","events_url":"https://api.github.com/users/mikedanese/events{/privacy}","received_events_url":"https://api.github.com/users/mikedanese/received_events","type":"User","site_admin":false},"milestone":null,"comments":27,"created_at":"2016-02-17T01:57:47Z","updated_at":"2016-03-04T23:38:29Z","closed_at":null,"body":"https://cloud.google.com/console/storage/kubernetes-jenkins/logs/kubernetes-e2e-gce/11727/\r\n```\r\n17:20:37 [BeforeEach] Pods\r\n17:20:37   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework.go:82\r\n17:20:37 STEP: Creating a kubernetes client\r\n17:20:37 Feb 16 17:17:59.419: INFO: >>> testContext.KubeConfig: /var/lib/jenkins/jobs/kubernetes-e2e-gce/workspace/.kube/config\r\n17:20:37 \r\n17:20:37 STEP: Building a namespace api object\r\n17:20:37 Feb 16 17:17:59.928: INFO: Waiting up to 2m0s for service account default to be provisioned in ns e2e-tests-pods-228zl\r\n17:20:37 Feb 16 17:17:59.934: INFO: Get service account default in ns e2e-tests-pods-228zl failed, ignoring for 2s: serviceaccounts \"default\" not found\r\n17:20:37 Feb 16 17:18:01.961: INFO: Service account default in ns e2e-tests-pods-228zl had 0 secrets, ignoring for 2s: <nil>\r\n17:20:37 Feb 16 17:18:03.964: INFO: Service account default in ns e2e-tests-pods-228zl with secrets found. (4.035770289s)\r\n17:20:37 STEP: Waiting for a default service account to be provisioned in namespace\r\n17:20:37 Feb 16 17:18:03.964: INFO: Waiting up to 2m0s for service account default to be provisioned in ns e2e-tests-pods-228zl\r\n17:20:37 Feb 16 17:18:03.966: INFO: Service account default in ns e2e-tests-pods-228zl with secrets found. (2.487832ms)\r\n17:20:37 [It] should be submitted and removed [Conformance]\r\n17:20:37   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:385\r\n17:20:37 STEP: creating the pod\r\n17:20:37 STEP: setting up watch\r\n17:20:37 STEP: submitting the pod to kubernetes\r\n17:20:37 STEP: verifying the pod is in kubernetes\r\n17:20:37 STEP: verifying pod creation was observed\r\n17:20:37 Feb 16 17:18:04.050: INFO: Waiting up to 5m0s for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 status to be running\r\n17:20:37 Feb 16 17:18:04.058: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (7.588859ms elapsed)\r\n17:20:37 Feb 16 17:18:06.063: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (2.013082419s elapsed)\r\n17:20:37 Feb 16 17:18:08.067: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (4.01691102s elapsed)\r\n17:20:37 Feb 16 17:18:10.074: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (6.024024849s elapsed)\r\n17:20:37 Feb 16 17:18:12.078: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (8.028132847s elapsed)\r\n17:20:37 Feb 16 17:18:14.082: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (10.031899756s elapsed)\r\n17:20:37 Feb 16 17:18:16.086: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (12.035981389s elapsed)\r\n17:20:37 Feb 16 17:18:18.090: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (14.039910403s elapsed)\r\n17:20:37 Feb 16 17:18:20.094: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (16.043625894s elapsed)\r\n17:20:37 Feb 16 17:18:22.099: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (18.048403899s elapsed)\r\n17:20:37 Feb 16 17:18:24.109: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (20.05852669s elapsed)\r\n17:20:37 Feb 16 17:18:26.113: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (22.062561758s elapsed)\r\n17:20:37 Feb 16 17:18:28.117: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (24.066553297s elapsed)\r\n17:20:37 Feb 16 17:18:30.120: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (26.069736255s elapsed)\r\n17:20:37 Feb 16 17:18:32.124: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (28.073592029s elapsed)\r\n17:20:37 Feb 16 17:18:34.128: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (30.077226774s elapsed)\r\n17:20:37 Feb 16 17:18:36.131: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (32.081156348s elapsed)\r\n17:20:37 Feb 16 17:18:38.135: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (34.084971958s elapsed)\r\n17:20:37 Feb 16 17:18:40.145: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (36.095070914s elapsed)\r\n17:20:37 Feb 16 17:18:42.150: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (38.099455618s elapsed)\r\n17:20:37 Feb 16 17:18:44.153: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (40.103094673s elapsed)\r\n17:20:37 Feb 16 17:18:46.160: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (42.110041311s elapsed)\r\n17:20:37 Feb 16 17:18:48.166: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (44.115844855s elapsed)\r\n17:20:37 Feb 16 17:18:50.170: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (46.119387359s elapsed)\r\n17:20:37 Feb 16 17:18:52.222: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (48.171409157s elapsed)\r\n17:20:37 Feb 16 17:18:54.226: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (50.175359544s elapsed)\r\n17:20:37 Feb 16 17:18:56.230: INFO: Waiting for pod pod-update-4b09d888-d514-11e5-be28-42010af01555 in namespace 'e2e-tests-pods-228zl' status to be 'running'(found phase: \"Pending\", readiness: false) (52.179725987s elapsed)\r\n17:20:37 Feb 16 17:18:58.238: INFO: Found pod 'pod-update-4b09d888-d514-11e5-be28-42010af01555' on node 'e2e-gce-minion-f1hj'\r\n17:20:37 STEP: deleting the pod gracefully\r\n17:20:37 STEP: verifying pod deletion was observed\r\n17:20:37 [AfterEach] Pods\r\n17:20:37   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework.go:83\r\n17:20:37 STEP: Collecting events from namespace \"e2e-tests-pods-228zl\".\r\n17:20:37 Feb 16 17:19:28.349: INFO: event for pod-update-4b09d888-d514-11e5-be28-42010af01555: {default-scheduler } Scheduled: Successfully assigned pod-update-4b09d888-d514-11e5-be28-42010af01555 to e2e-gce-minion-f1hj\r\n17:20:37 Feb 16 17:19:28.349: INFO: event for pod-update-4b09d888-d514-11e5-be28-42010af01555: {kubelet e2e-gce-minion-f1hj} Pulling: pulling image \"gcr.io/google_containers/nginx:1.7.9\"\r\n17:20:37 Feb 16 17:19:28.349: INFO: event for pod-update-4b09d888-d514-11e5-be28-42010af01555: {kubelet e2e-gce-minion-f1hj} Pulled: Successfully pulled image \"gcr.io/google_containers/nginx:1.7.9\"\r\n17:20:37 Feb 16 17:19:28.349: INFO: event for pod-update-4b09d888-d514-11e5-be28-42010af01555: {kubelet e2e-gce-minion-f1hj} Created: Created container with docker id 6487ce8f8594\r\n17:20:37 Feb 16 17:19:28.349: INFO: event for pod-update-4b09d888-d514-11e5-be28-42010af01555: {kubelet e2e-gce-minion-f1hj} Started: Started container with docker id 6487ce8f8594\r\n17:20:37 Feb 16 17:19:28.349: INFO: event for pod-update-4b09d888-d514-11e5-be28-42010af01555: {kubelet e2e-gce-minion-f1hj} Killing: Killing container with docker id 6487ce8f8594: Need to kill pod.\r\n17:20:37 Feb 16 17:19:28.399: INFO: POD                                                                    NODE                 PHASE      GRACE  CONDITIONS\r\n17:20:37 Feb 16 17:19:28.399: INFO: test-host-network-pod                                                  e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:20 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: test-pod                                                               e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:09 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: pod-6c8791ee-d514-11e5-bd47-42010af01555                               e2e-gce-minion-f1hj  Pending           []\r\n17:20:37 Feb 16 17:19:28.399: INFO: fail-once-local-1m30h                                                  e2e-gce-minion-oerq  Succeeded         [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:10 -0800 PST PodCompleted }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: fail-once-local-4uhsp                                                  e2e-gce-minion-cnua  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:25 -0800 PST ContainersNotReady containers with unready status: [c]}]\r\n17:20:37 Feb 16 17:19:28.399: INFO: fail-once-local-8pr8w                                                  e2e-gce-minion-oerq  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:23 -0800 PST ContainersNotReady containers with unready status: [c]}]\r\n17:20:37 Feb 16 17:19:28.399: INFO: fail-once-local-phkx0                                                  e2e-gce-minion-oerq  Succeeded         [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:10 -0800 PST PodCompleted }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: foo-12zjn                                                              e2e-gce-minion-cnua  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:23 -0800 PST ContainersNotReady containers with unready status: [c]}]\r\n17:20:37 Feb 16 17:19:28.399: INFO: foo-pw3gl                                                              e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:28 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: redis-master-9xc5q                                                     e2e-gce-minion-cnua  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:26 -0800 PST ContainersNotReady containers with unready status: [redis-master]}]\r\n17:20:37 Feb 16 17:19:28.399: INFO: update-demo-nautilus-cuevs                                             e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:08 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: nginx                                                                  e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:21 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: run-test-talfe                                                         e2e-gce-minion-cnua  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:25 -0800 PST ContainersNotReady containers with unready status: [run-test]}]\r\n17:20:37 Feb 16 17:19:28.399: INFO: goproxy                                                                e2e-gce-minion-cnua  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:22 -0800 PST ContainersNotReady containers with unready status: [goproxy]}]\r\n17:20:37 Feb 16 17:19:28.399: INFO: netexec                                                                e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:20 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: nginx                                                                  e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:16 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.399: INFO: frontend-2xvkm                                                         e2e-gce-minion-cnua  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:18:06 -0800 PST ContainersNotReady containers with unready status: [php-redis]}]\r\n17:20:37 Feb 16 17:19:28.399: INFO: frontend-e3whh                                                         e2e-gce-minion-f1hj  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:18:03 -0800 PST ContainersNotReady containers with unready status: [php-redis]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: frontend-lsp0i                                                         e2e-gce-minion-oerq  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:18:03 -0800 PST ContainersNotReady containers with unready status: [php-redis]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: redis-master-2nxao                                                     e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:11 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: redis-slave-cz84r                                                      e2e-gce-minion-f1hj  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:18:04 -0800 PST ContainersNotReady containers with unready status: [slave]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: redis-slave-huzyp                                                      e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:22 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-044zj                   e2e-gce-minion-f1hj  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:04 -0800 PST ContainersNotReady containers with unready status: [cleanup30-67b6a779-d514-11e5-b81b-42010af01555]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0f7zj                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0n36r                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0nrcr                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:04 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0t3ly                   e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:25 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-2pdte                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:00 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-3s782                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-54tb7                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:03 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-62z11                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:04 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-6y2ib                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:03 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-76jwb                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-7f1qk                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:02 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-bn62x                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:02 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-eaf7t                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-egwet                   e2e-gce-minion-f1hj  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:19 -0800 PST ContainersNotReady containers with unready status: [cleanup30-67b6a779-d514-11e5-b81b-42010af01555]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-g7ab7                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:00 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-jhcqz                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:05 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-kqswk                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:04 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-lyoph                   e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:19 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-m8ec9                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:00 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-n0srw                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-n2u5i                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-pegbp                   e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:26 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-pjdbq                   e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:02 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-ptaym                   e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:19 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-t2gb0                   e2e-gce-minion-f1hj  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:19 -0800 PST ContainersNotReady containers with unready status: [cleanup30-67b6a779-d514-11e5-b81b-42010af01555]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-trzt9                   e2e-gce-minion-f1hj  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:10 -0800 PST ContainersNotReady containers with unready status: [cleanup30-67b6a779-d514-11e5-b81b-42010af01555]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-y78g3                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:00 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-yak4s                   e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:19 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-yv2ed                   e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:00 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: pd-test-5c243ad2-d514-11e5-9ccd-42010af01555                           e2e-gce-minion-f1hj  Pending           []\r\n17:20:37 Feb 16 17:19:28.400: INFO: liveness-exec                                                          e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:21 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: pod-update-7237a2de-d514-11e5-815e-42010af01555                        e2e-gce-minion-f1hj  Pending           []\r\n17:20:37 Feb 16 17:19:28.400: INFO: liveness-http                                                          e2e-gce-minion-f1hj  Pending           []\r\n17:20:37 Feb 16 17:19:28.400: INFO: pod-update-activedeadlineseconds-6b11b006-d514-11e5-a8e7-42010af01555  e2e-gce-minion-f1hj  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:19 -0800 PST ContainersNotReady containers with unready status: [nginx]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: proxy-service-ag3av-9g3v4                                              e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:18:12 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: mutability-test-7egaa                                                  e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:12 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: mutability-test-i9n69                                                  e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:01 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: service1-6dey7                                                         e2e-gce-minion-oerq  Pending           []\r\n17:20:37 Feb 16 17:19:28.400: INFO: service1-86ttb                                                         e2e-gce-minion-f1hj  Pending           []\r\n17:20:37 Feb 16 17:19:28.400: INFO: service1-b0502                                                         e2e-gce-minion-cnua  Pending           [{Ready False 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:19:26 -0800 PST ContainersNotReady containers with unready status: [service1]}]\r\n17:20:37 Feb 16 17:19:28.400: INFO: elasticsearch-logging-v1-36pa5                                         e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:17:39 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: elasticsearch-logging-v1-bhap4                                         e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:17:39 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: fluentd-elasticsearch-e2e-gce-minion-cnua                              e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:16:20 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: fluentd-elasticsearch-e2e-gce-minion-f1hj                              e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:17:44 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: fluentd-elasticsearch-e2e-gce-minion-oerq                              e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:16:37 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: heapster-v14-9lkrx                                                     e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:17:07 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: kibana-logging-v1-8xspg                                                e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:18:09 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: kube-dns-v10-i7gih                                                     e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:17:58 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: kube-proxy-e2e-gce-minion-cnua                                         e2e-gce-minion-cnua  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:16:09 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: kube-proxy-e2e-gce-minion-f1hj                                         e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:16:22 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: kube-proxy-e2e-gce-minion-oerq                                         e2e-gce-minion-oerq  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:16:24 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: kubernetes-dashboard-v0.1.0-7nzy4                                      e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:16:47 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: l7-lb-controller-x29zb                                                 e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:17:45 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: monitoring-influxdb-grafana-v3-ybudt                                   e2e-gce-minion-f1hj  Running           [{Ready True 0001-01-01 00:00:00 +0000 UTC 2016-02-16 17:17:49 -0800 PST  }]\r\n17:20:37 Feb 16 17:19:28.400: INFO: \r\n17:20:37 Feb 16 17:19:28.405: INFO: \r\n17:20:37 Logging node info for node e2e-gce-minion-cnua\r\n17:20:37 Feb 16 17:19:28.413: INFO: Node Info: &{{ } {e2e-gce-minion-cnua   /api/v1/nodes/e2e-gce-minion-cnua f30b091b-d513-11e5-9301-42010af00002 5026 0 2016-02-16 17:15:36 -0800 PST <nil> <nil> map[failure-domain.alpha.kubernetes.io/region:us-central1 failure-domain.alpha.kubernetes.io/zone:us-central1-f kubernetes.io/hostname:e2e-gce-minion-cnua beta.kubernetes.io/instance-type:n1-standard-2] map[]} {10.245.0.0/24 6659199225896706916 gce://k8s-jkns-e2e-gce/us-central1-f/e2e-gce-minion-cnua false} {map[memory:{7864139776.000 BinarySI} pods:{110.000 DecimalSI} cpu:{2.000 DecimalSI}] map[pods:{110.000 DecimalSI} cpu:{2.000 DecimalSI} memory:{7864139776.000 BinarySI}]  [{OutOfDisk False 2016-02-16 17:19:22 -0800 PST 2016-02-16 17:15:36 -0800 PST KubeletHasSufficientDisk kubelet has sufficient disk space available} {Ready True 2016-02-16 17:19:22 -0800 PST 2016-02-16 17:15:56 -0800 PST KubeletReady kubelet is posting ready status}] [{InternalIP 10.240.0.5} {ExternalIP 104.154.18.24}] {{10250}} { 7D5F6764-2797-BFC0-F7D3-0FE5B513157B 0dc1c892-6b4f-4c8c-84c2-cd75f8125fd8 3.16.0-4-amd64 Debian GNU/Linux 7 (wheezy) docker://1.9.1 v1.2.0-alpha.7.955+3616b4bfecc9a3 v1.2.0-alpha.7.955+3616b4bfecc9a3} [{[gcr.io/google_containers/kube-proxy:c333dcf386ec361c6b1a1e74941e87ab] 164583261} {[gcr.io/google_containers/nettest:1.7] 24051275} {[nginx:latest] 134581595} {[gcr.io/google_containers/fluentd-elasticsearch:1.14] 562034622} {[gcr.io/google_containers/netexec:1.4] 7297019} {[redis:latest] 151289204} {[gcr.io/google_containers/busybox:1.24] 1113554} {[gcr.io/google_containers/mounttest:0.6] 2084693} {[gcr.io/google_containers/hostexec:1.2] 13209617} {[gcr.io/google_containers/mounttest-user:0.3] 1718853} {[gcr.io/google_containers/mounttest:0.5] 1718853} {[gcr.io/google_containers/pause:2.0] 350164} {[<none>:<none>] 328223859} {[gcr.io/google_containers/pause:0.8.0] 241656} {[gcr.io/google_containers/eptest:0.1] 2970692} {[gcr.io/google_containers/serve_hostname:1.1] 4522409} {[gcr.io/google_containers/update-demo:nautilus] 4555533} {[gcr.io/google_containers/nginx:1.7.9] 91664166}]}}\r\n17:20:37 Feb 16 17:19:28.413: INFO: \r\n17:20:37 Logging kubelet events for node e2e-gce-minion-cnua\r\n17:20:37 Feb 16 17:19:28.422: INFO: \r\n17:20:37 Logging pods the kubelet thinks is on node e2e-gce-minion-cnua\r\n17:20:37 Feb 16 17:19:28.433: INFO: frontend-2xvkm started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-7f1qk started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: kube-proxy-e2e-gce-minion-cnua started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: run-test-talfe started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: netexec started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: redis-master-2nxao started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-pjdbq started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: foo-12zjn started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: mutability-test-7egaa started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: nginx started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-6y2ib started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: fluentd-elasticsearch-e2e-gce-minion-cnua started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: fail-once-local-4uhsp started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: pd-test-685ba56b-d514-11e5-98a7-42010af01555 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: pd-test-5c243a53-d514-11e5-9ccd-42010af01555 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-76jwb started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-jhcqz started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: service1-b0502 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: update-demo-nautilus-cuevs started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0nrcr started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: nginx started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-kqswk started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-62z11 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: redis-master-9xc5q started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: liveness-exec started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: hostexec started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: foo-pw3gl started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: goproxy started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.433: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-54tb7 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:28.967: INFO: ERROR kubelet_docker_errors{operation_type=\"info\"} => 1 @[0]\r\n17:20:37 Feb 16 17:19:28.967: INFO: ERROR kubelet_docker_errors{operation_type=\"inspect_image\"} => 29 @[0]\r\n17:20:37 Feb 16 17:19:28.967: INFO: ERROR kubelet_docker_errors{operation_type=\"list_containers\"} => 63 @[0]\r\n17:20:37 Feb 16 17:19:28.967: INFO: ERROR kubelet_docker_errors{operation_type=\"list_images\"} => 15 @[0]\r\n17:20:37 Feb 16 17:19:28.967: INFO: ERROR kubelet_docker_errors{operation_type=\"pull_image\"} => 1 @[0]\r\n17:20:37 Feb 16 17:19:28.967: INFO: ERROR kubelet_docker_errors{operation_type=\"stop_container\"} => 8 @[0]\r\n17:20:37 Feb 16 17:19:28.967: INFO: ERROR kubelet_docker_errors{operation_type=\"version\"} => 54 @[0]\r\n17:20:37 Feb 16 17:19:28.967: INFO: \r\n17:20:37 Latency metrics for node e2e-gce-minion-cnua\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:1m8.94002s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.99 Latency:1m5.11241s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.99 Latency:1m4.709583s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation:SyncPod Method:container_manager_latency_microseconds Quantile:0.99 Latency:1m1.584456s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.9 Latency:1m1.234997s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation:pull_image Method:docker_operations_latency_microseconds Quantile:0.99 Latency:1m1.091161s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation:pull_image Method:docker_operations_latency_microseconds Quantile:0.9 Latency:57.517177s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:54.763768s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.9 Latency:37.960341s}\r\n17:20:37 Feb 16 17:19:28.967: INFO: \r\n17:20:37 Logging node info for node e2e-gce-minion-f1hj\r\n17:20:37 Feb 16 17:19:28.971: INFO: Node Info: &{{ } {e2e-gce-minion-f1hj   /api/v1/nodes/e2e-gce-minion-f1hj f25621f5-d513-11e5-9301-42010af00002 5000 0 2016-02-16 17:15:35 -0800 PST <nil> <nil> map[beta.kubernetes.io/instance-type:n1-standard-2 failure-domain.alpha.kubernetes.io/region:us-central1 failure-domain.alpha.kubernetes.io/zone:us-central1-f kubernetes.io/hostname:e2e-gce-minion-f1hj] map[]} {10.245.2.0/24 512911018671407553 gce://k8s-jkns-e2e-gce/us-central1-f/e2e-gce-minion-f1hj false} {map[cpu:{2.000 DecimalSI} memory:{7864139776.000 BinarySI} pods:{110.000 DecimalSI}] map[pods:{110.000 DecimalSI} cpu:{2.000 DecimalSI} memory:{7864139776.000 BinarySI}]  [{OutOfDisk False 2016-02-16 17:19:20 -0800 PST 2016-02-16 17:15:35 -0800 PST KubeletHasSufficientDisk kubelet has sufficient disk space available} {Ready True 2016-02-16 17:19:20 -0800 PST 2016-02-16 17:15:55 -0800 PST KubeletReady kubelet is posting ready status}] [{InternalIP 10.240.0.4} {ExternalIP 146.148.45.47}] {{10250}} { 885AD8CB-94A5-A7D3-2251-23478E25C5DB cb30f46b-70fc-4381-84f5-259ba3b22d57 3.16.0-4-amd64 Debian GNU/Linux 7 (wheezy) docker://1.9.1 v1.2.0-alpha.7.955+3616b4bfecc9a3 v1.2.0-alpha.7.955+3616b4bfecc9a3} [{[gcr.io/google_containers/kube-proxy:c333dcf386ec361c6b1a1e74941e87ab] 164583261} {[gcr.io/google_containers/heapster:v0.20.0-alpha6] 85929942} {[gcr.io/google_containers/nettest:1.7] 24051275} {[gcr.io/google_containers/fluentd-elasticsearch:1.14] 562034622} {[gcr.io/google_containers/kubernetes-dashboard-amd64:v0.1.0] 35409795} {[gcr.io/google_containers/netexec:1.4] 7297019} {[gcr.io/google_containers/glbc:0.5.2] 226351998} {[gcr.io/google_containers/busybox:1.24] 1113554} {[gcr.io/google_containers/kube2sky:1.12] 24482187} {[gcr.io/google_containers/elasticsearch:1.8] 410989305} {[gcr.io/google_containers/defaultbackend:1.0] 7513643} {[gcr.io/google_containers/skydns:2015-10-13-8c72f8c] 40551394} {[gcr.io/google_containers/heapster_grafana:v2.1.1] 206641132} {[gcr.io/google_containers/pause:2.0] 350164} {[gcr.io/google_containers/heapster_influxdb:v0.5] 251005705} {[gcr.io/google_samples/gb-frontend:v3] 509707703} {[gcr.io/google_samples/gb-redisslave:v1] 109508753} {[gcr.io/google_containers/exechealthz:1.0] 7099444} {[gcr.io/google_containers/kibana:1.3] 396897764} {[gcr.io/google_containers/etcd:2.0.9] 12819040} {[gcr.io/google_containers/pause:0.8.0] 241656} {[gcr.io/google_containers/eptest:0.1] 2970692} {[gcr.io/google_containers/nginx:1.7.9] 91664166}]}}\r\n17:20:37 Feb 16 17:19:28.971: INFO: \r\n17:20:37 Logging kubelet events for node e2e-gce-minion-f1hj\r\n17:20:37 Feb 16 17:19:28.979: INFO: \r\n17:20:37 Logging pods the kubelet thinks is on node e2e-gce-minion-f1hj\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-lyoph started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: monitoring-influxdb-grafana-v3-ybudt started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: pod-update-7237a2de-d514-11e5-815e-42010af01555 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: pod-6c8791ee-d514-11e5-bd47-42010af01555 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: frontend-e3whh started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-yak4s started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: liveness-http started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: pod-update-activedeadlineseconds-6b11b006-d514-11e5-a8e7-42010af01555 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-egwet started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: kube-dns-v10-i7gih started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: l7-lb-controller-x29zb started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: fluentd-elasticsearch-e2e-gce-minion-f1hj started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: kubernetes-dashboard-v0.1.0-7nzy4 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-t2gb0 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: redis-slave-cz84r started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-044zj started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-ptaym started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: kube-proxy-e2e-gce-minion-f1hj started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: kibana-logging-v1-8xspg started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: elasticsearch-logging-v1-bhap4 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: elasticsearch-logging-v1-36pa5 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: heapster-v14-9lkrx started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-pegbp started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0t3ly started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-trzt9 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:31.204: INFO: mutability-test-i9n69 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.035: INFO: ERROR kubelet_docker_errors{operation_type=\"info\"} => 1 @[0]\r\n17:20:37 Feb 16 17:19:32.035: INFO: ERROR kubelet_docker_errors{operation_type=\"inspect_image\"} => 36 @[0]\r\n17:20:37 Feb 16 17:19:32.035: INFO: ERROR kubelet_docker_errors{operation_type=\"list_containers\"} => 63 @[0]\r\n17:20:37 Feb 16 17:19:32.035: INFO: ERROR kubelet_docker_errors{operation_type=\"list_images\"} => 14 @[0]\r\n17:20:37 Feb 16 17:19:32.035: INFO: ERROR kubelet_docker_errors{operation_type=\"pull_image\"} => 1 @[0]\r\n17:20:37 Feb 16 17:19:32.035: INFO: ERROR kubelet_docker_errors{operation_type=\"stop_container\"} => 2 @[0]\r\n17:20:37 Feb 16 17:19:32.035: INFO: ERROR kubelet_docker_errors{operation_type=\"version\"} => 52 @[0]\r\n17:20:37 Feb 16 17:19:32.036: INFO: \r\n17:20:37 Latency metrics for node e2e-gce-minion-f1hj\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:1m54.115608s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.99 Latency:1m50.981142s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:SyncPod Method:container_manager_latency_microseconds Quantile:0.99 Latency:1m50.372212s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.9 Latency:1m45.795401s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.9 Latency:1m44.285608s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:pull_image Method:docker_operations_latency_microseconds Quantile:0.99 Latency:1m41.79396s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:1m38.621752s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:pull_image Method:docker_operations_latency_microseconds Quantile:0.9 Latency:1m28.251474s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:SyncPod Method:container_manager_latency_microseconds Quantile:0.9 Latency:1m7.561168s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.99 Latency:1m5.036688s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.9 Latency:52.357767s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.5 Latency:40.991666s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation:pull_image Method:docker_operations_latency_microseconds Quantile:0.5 Latency:28.490731s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.9 Latency:26.823067s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.5 Latency:26.682908s}\r\n17:20:37 Feb 16 17:19:32.036: INFO: \r\n17:20:37 Logging node info for node e2e-gce-minion-oerq\r\n17:20:37 Feb 16 17:19:32.040: INFO: Node Info: &{{ } {e2e-gce-minion-oerq   /api/v1/nodes/e2e-gce-minion-oerq f4b1cb9b-d513-11e5-9301-42010af00002 5101 0 2016-02-16 17:15:39 -0800 PST <nil> <nil> map[beta.kubernetes.io/instance-type:n1-standard-2 failure-domain.alpha.kubernetes.io/region:us-central1 failure-domain.alpha.kubernetes.io/zone:us-central1-f kubernetes.io/hostname:e2e-gce-minion-oerq] map[]} {10.245.1.0/24 7824052894667841549 gce://k8s-jkns-e2e-gce/us-central1-f/e2e-gce-minion-oerq false} {map[cpu:{2.000 DecimalSI} memory:{7864139776.000 BinarySI} pods:{110.000 DecimalSI}] map[memory:{7864139776.000 BinarySI} pods:{110.000 DecimalSI} cpu:{2.000 DecimalSI}]  [{OutOfDisk False 2016-02-16 17:19:25 -0800 PST 2016-02-16 17:15:39 -0800 PST KubeletHasSufficientDisk kubelet has sufficient disk space available} {Ready True 2016-02-16 17:19:25 -0800 PST 2016-02-16 17:16:09 -0800 PST KubeletReady kubelet is posting ready status}] [{InternalIP 10.240.0.3} {ExternalIP 104.197.37.181}] {{10250}} { 9909F2DF-988C-689B-E589-B9B616145386 3dd4d725-14ae-43aa-ac3f-af21d743ef96 3.16.0-4-amd64 Debian GNU/Linux 7 (wheezy) docker://1.9.1 v1.2.0-alpha.7.955+3616b4bfecc9a3 v1.2.0-alpha.7.955+3616b4bfecc9a3} [{[gcr.io/google_containers/kube-proxy:c333dcf386ec361c6b1a1e74941e87ab] 164583261} {[busybox:latest] 1113554} {[gcr.io/google_containers/nettest:1.7] 24051275} {[nginx:latest] 134581595} {[gcr.io/google_containers/fluentd-elasticsearch:1.14] 562034622} {[gcr.io/google_containers/netexec:1.4] 7297019} {[gcr.io/google_containers/busybox:1.24] 1113554} {[gcr.io/google_containers/mounttest:0.6] 2084693} {[gcr.io/google_containers/mounttest-user:0.3] 1718853} {[gcr.io/google_containers/mounttest:0.5] 1718853} {[gcr.io/google_containers/pause:2.0] 350164} {[gcr.io/google_containers/porter:cd5cb5791ebaa8641955f0e8c2a9bed669b1eaab] 5010921} {[<none>:<none>] 509705816} {[gcr.io/google_samples/gb-redisslave:v1] 109508753} {[gcr.io/google_containers/pause:0.8.0] 241656} {[gcr.io/google_containers/serve_hostname:1.1] 4522409} {[gcr.io/google_containers/update-demo:nautilus] 4555533}]}}\r\n17:20:37 Feb 16 17:19:32.040: INFO: \r\n17:20:37 Logging kubelet events for node e2e-gce-minion-oerq\r\n17:20:37 Feb 16 17:19:32.049: INFO: \r\n17:20:37 Logging pods the kubelet thinks is on node e2e-gce-minion-oerq\r\n17:20:37 Feb 16 17:19:32.060: INFO: test-host-network-pod started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: foo-j4pgz started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-n0srw started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-m8ec9 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-y78g3 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: proxy-service-ag3av-9g3v4 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: scale-down-2si2f started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: frontend-lsp0i started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: test-pod started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: foo-b9ld0 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-eaf7t started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-3s782 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-n2u5i started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0n36r started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-0f7zj started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: tester started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: fluentd-elasticsearch-e2e-gce-minion-oerq started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-g7ab7 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-2pdte started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-bn62x started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: kube-proxy-e2e-gce-minion-oerq started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: cleanup30-67b6a779-d514-11e5-b81b-42010af01555-yv2ed started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: redis-slave-huzyp started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.060: INFO: service1-6dey7 started at <nil> (0 container statuses recorded)\r\n17:20:37 Feb 16 17:19:32.455: INFO: ERROR kubelet_docker_errors{operation_type=\"info\"} => 1 @[0]\r\n17:20:37 Feb 16 17:19:32.455: INFO: ERROR kubelet_docker_errors{operation_type=\"inspect_image\"} => 24 @[0]\r\n17:20:37 Feb 16 17:19:32.455: INFO: ERROR kubelet_docker_errors{operation_type=\"list_containers\"} => 63 @[0]\r\n17:20:37 Feb 16 17:19:32.455: INFO: ERROR kubelet_docker_errors{operation_type=\"list_images\"} => 14 @[0]\r\n17:20:37 Feb 16 17:19:32.455: INFO: ERROR kubelet_docker_errors{operation_type=\"pull_image\"} => 1 @[0]\r\n17:20:37 Feb 16 17:19:32.455: INFO: ERROR kubelet_docker_errors{operation_type=\"stop_container\"} => 16 @[0]\r\n17:20:37 Feb 16 17:19:32.455: INFO: ERROR kubelet_docker_errors{operation_type=\"version\"} => 52 @[0]\r\n17:20:37 Feb 16 17:19:32.456: INFO: \r\n17:20:37 Latency metrics for node e2e-gce-minion-oerq\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:1m20.587562s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.99 Latency:1m18.44162s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation:pull_image Method:docker_operations_latency_microseconds Quantile:0.99 Latency:1m13.657342s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.99 Latency:1m5.077098s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation:SyncPod Method:container_manager_latency_microseconds Quantile:0.99 Latency:59.571551s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:56.872256s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation:pull_image Method:docker_operations_latency_microseconds Quantile:0.9 Latency:54.558065s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation:create Method:pod_worker_latency_microseconds Quantile:0.9 Latency:45.357772s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.9 Latency:29.01991s}\r\n17:20:37 Feb 16 17:19:32.456: INFO: Waiting up to 1m0s for all nodes to be ready\r\n17:20:37 STEP: Destroying namespace \"e2e-tests-pods-228zl\" for this suite.\r\n17:20:37 \r\n17:20:37 \r\n17:20:37 • Failure [158.056 seconds]\r\n17:20:37 Pods\r\n17:20:37 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:1263\r\n17:20:37   should be submitted and removed [Conformance] [It]\r\n17:20:37   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:385\r\n17:20:37 \r\n17:20:37   Failed to observe pod deletion\r\n17:20:37 \r\n17:20:37   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:372\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/192525357","html_url":"https://github.com/kubernetes/kubernetes/issues/21367#issuecomment-192525357","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21367","id":192525357,"user":{"login":"mikedanese","id":1787169,"avatar_url":"https://avatars.githubusercontent.com/u/1787169?v=3","gravatar_id":"","url":"https://api.github.com/users/mikedanese","html_url":"https://github.com/mikedanese","followers_url":"https://api.github.com/users/mikedanese/followers","following_url":"https://api.github.com/users/mikedanese/following{/other_user}","gists_url":"https://api.github.com/users/mikedanese/gists{/gist_id}","starred_url":"https://api.github.com/users/mikedanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikedanese/subscriptions","organizations_url":"https://api.github.com/users/mikedanese/orgs","repos_url":"https://api.github.com/users/mikedanese/repos","events_url":"https://api.github.com/users/mikedanese/events{/privacy}","received_events_url":"https://api.github.com/users/mikedanese/received_events","type":"User","site_admin":false},"created_at":"2016-03-04T23:38:28Z","updated_at":"2016-03-04T23:38:28Z","body":"I submitted #22565 which will allow us to wait for kubelet to acknowledge that it has received the deletion request before we start the timer."}}