{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20500","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20500/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20500/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20500/events","html_url":"https://github.com/kubernetes/kubernetes/issues/20500","id":130770784,"number":20500,"title":"integration flake: runServiceTest \"FAILED: unexpected endpoints: timed out waiting for the condition\"","user":{"login":"jlowdermilk","id":2101035,"avatar_url":"https://avatars.githubusercontent.com/u/2101035?v=3","gravatar_id":"","url":"https://api.github.com/users/jlowdermilk","html_url":"https://github.com/jlowdermilk","followers_url":"https://api.github.com/users/jlowdermilk/followers","following_url":"https://api.github.com/users/jlowdermilk/following{/other_user}","gists_url":"https://api.github.com/users/jlowdermilk/gists{/gist_id}","starred_url":"https://api.github.com/users/jlowdermilk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jlowdermilk/subscriptions","organizations_url":"https://api.github.com/users/jlowdermilk/orgs","repos_url":"https://api.github.com/users/jlowdermilk/repos","events_url":"https://api.github.com/users/jlowdermilk/events{/privacy}","received_events_url":"https://api.github.com/users/jlowdermilk/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/flake","name":"kind/flake","color":"f7c6c7"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"cjcullen","id":7154008,"avatar_url":"https://avatars.githubusercontent.com/u/7154008?v=3","gravatar_id":"","url":"https://api.github.com/users/cjcullen","html_url":"https://github.com/cjcullen","followers_url":"https://api.github.com/users/cjcullen/followers","following_url":"https://api.github.com/users/cjcullen/following{/other_user}","gists_url":"https://api.github.com/users/cjcullen/gists{/gist_id}","starred_url":"https://api.github.com/users/cjcullen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjcullen/subscriptions","organizations_url":"https://api.github.com/users/cjcullen/orgs","repos_url":"https://api.github.com/users/cjcullen/repos","events_url":"https://api.github.com/users/cjcullen/events{/privacy}","received_events_url":"https://api.github.com/users/cjcullen/received_events","type":"User","site_admin":false},"milestone":null,"comments":10,"created_at":"2016-02-02T18:22:35Z","updated_at":"2016-02-04T20:45:52Z","closed_at":null,"body":"Failed test logs:\r\n[kubernetes-test-go-release-1.1/3464](https://storage.cloud.google.com/kubernetes-jenkins/logs/kubernetes-test-go-release-1.1/3464)\r\n[kubernetes-test-go-release-1.1/3467](https://storage.cloud.google.com/kubernetes-jenkins/logs/kubernetes-test-go-release-1.1/3467)\r\n\r\nExcerpt:\r\n```\r\n...\r\n02:31:51 I0202 10:31:51.061294   11528 kubelet.go:2747] Generating status for \"container-vm-guestbook-pod-spec-127.0.0.1_default\"\r\n02:31:51 I0202 10:31:51.061328   11528 manager.go:357] Container inspect result: {ID:/k8s_nginx.72c3fedf_nginx-controller-2jgc8_test_79b0bc33-c997-11e5-8de7-0242ac110003_c4d89e98 Created:0001-01-01 00:00:00 +0000 UTC Path: Args:[] Config:0xc208abf860 State:{Running:true Paused:false Restarting:false OOMKilled:false Pid:11528 ExitCode:0 Error: StartedAt:2016-02-02 10:27:01.068093205 +0000 UTC FinishedAt:0001-01-01 00:00:00 +0000 UTC} Image: Node:<nil> NetworkSettings:0xc208924e00 SysInitPath: ResolvConfPath: HostnamePath: HostsPath: LogPath: Name:/k8s_nginx.72c3fedf_nginx-controller-2jgc8_test_79b0bc33-c997-11e5-8de7-0242ac110003_c4d89e98 Driver: Volumes:map[] VolumesRW:map[] HostConfig:0xc2092f5b00 ExecIDs:[] RestartCount:0 AppArmorProfile:}\r\n02:31:51 I0202 10:31:51.061391   11528 manager.go:357] Container inspect result: {ID:/k8s_nginx.72c3fedf_nginx-controller-2jgc8_test_79b0bc33-c997-11e5-8de7-0242ac110003_c4d89e98 Created:0001-01-01 00:00:00 +0000 UTC Path: Args:[] Config:0xc208abf860 State:{Running:true Paused:false Restarting:false OOMKilled:false Pid:11528 ExitCode:0 Error: StartedAt:2016-02-02 10:27:01.068093205 +0000 UTC FinishedAt:0001-01-01 00:00:00 +0000 UTC} Image: Node:<nil> NetworkSettings:0xc208924e00 SysInitPath: ResolvConfPath: HostnamePath: HostsPath: LogPath: Name:/k8s_nginx.72c3fedf_nginx-controller-2jgc8_test_79b0bc33-c997-11e5-8de7-0242ac110003_c4d89e98 Driver: Volumes:map[] VolumesRW:map[] HostConfig:0xc2092f5b00 ExecIDs:[] RestartCount:0 AppArmorProfile:}\r\n02:31:51 I0202 10:31:51.061450   11528 manager.go:357] Container inspect result: {ID:/k8s_nginx.72c3fedf_nginx-controller-2jgc8_test_79b0bc33-c997-11e5-8de7-0242ac110003_c4d89e98 Created:0001-01-01 00:00:00 +0000 UTC Path: Args:[] Config:0xc208abf860 State:{Running:true Paused:false Restarting:false OOMKilled:false Pid:11528 ExitCode:0 Error: StartedAt:2016-02-02 10:27:01.068093205 +0000 UTC FinishedAt:0001-01-01 00:00:00 +0000 UTC} Image: Node:<nil> NetworkSettings:0xc208924e00 SysInitPath: ResolvConfPath: HostnamePath: HostsPath: LogPath: Name:/k8s_nginx.72c3fedf_nginx-controller-2jgc8_test_79b0bc33-c997-11e5-8de7-0242ac110003_c4d89e98 Driver: Volumes:map[] VolumesRW:map[] HostConfig:0xc2092f5b00 ExecIDs:[] RestartCount:0 AppArmorProfile:}\r\n02:31:51 I0202 10:31:51.061590   11528 manager.go:156] Ignoring same status for pod \"container-vm-guestbook-pod-spec-127.0.0.1_default\", status: {Phase:Running Conditions:[{Type:Ready Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:0001-01-01 00:00:00 +0000 UTC Reason: Message:}] Message: Reason: HostIP:127.0.0.1 PodIP:1.2.3.4 StartTime:2016-02-02 10:26:46.567773694 +0000 UTC ContainerStatuses:[{Name:guestbook State:{Waiting:<nil> Running:0xc2093bcb40 Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:<nil>} Ready:true RestartCount:0 Image:testimage ImageID:docker:// ContainerID:docker:///k8s_guestbook.66e0ba8_container-vm-guestbook-pod-spec-127.0.0.1_default_03bc68c962f47e499c14a1fc11750d71_1924aa66} {Name:redis State:{Waiting:<nil> Running:0xc2093bcac0 Terminated:<nil>} LastTerminationState:{Waiting:<nil> Running:<nil> Terminated:<nil>} Ready:true RestartCount:0 Image:testimage ImageID:docker:// ContainerID:docker:///k8s_redis.5107f6a8_container-vm-guestbook-pod-spec-127.0.0.1_default_03bc68c962f47e499c14a1fc11750d71_139c0f0e}]}\r\n02:31:51 I0202 10:31:51.081105   11528 config.go:254] Setting pods for source file\r\n02:31:51 I0202 10:31:51.662718   11528 nodecontroller.go:611] Nodes ReadyCondition updated. Updating timestamp: {Capacity:map[memory:{Amount:0 Format:DecimalSI} pods:{Amount:32.000 Format:DecimalSI} cpu:{Amount:0 Format:DecimalSI}] Phase: Conditions:[{Type:OutOfDisk Status:False LastHeartbeatTime:2016-02-02 10:31:44 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletHasSufficientDisk Message:kubelet has sufficient disk space available} {Type:Ready Status:True LastHeartbeatTime:2016-02-02 10:31:44 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletReady Message:kubelet is posting ready status}] Addresses:[{Type:LegacyHostIP Address:127.0.0.1} {Type:InternalIP Address:127.0.0.1}] DaemonEndpoints:{KubeletEndpoint:{Port:10251}} NodeInfo:{MachineID: SystemUUID: BootID: KernelVersion: OsImage: ContainerRuntimeVersion:docker:// KubeletVersion:v1.1.8-beta.0.12+7e94fe062bc7c4 KubeProxyVersion:v1.1.8-beta.0.12+7e94fe062bc7c4}}\r\n02:31:51  vs {Capacity:map[memory:{Amount:0 Format:DecimalSI} pods:{Amount:32.000 Format:DecimalSI} cpu:{Amount:0 Format:DecimalSI}] Phase: Conditions:[{Type:OutOfDisk Status:False LastHeartbeatTime:2016-02-02 10:31:50 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletHasSufficientDisk Message:kubelet has sufficient disk space available} {Type:Ready Status:True LastHeartbeatTime:2016-02-02 10:31:50 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletReady Message:kubelet is posting ready status}] Addresses:[{Type:LegacyHostIP Address:127.0.0.1} {Type:InternalIP Address:127.0.0.1}] DaemonEndpoints:{KubeletEndpoint:{Port:10251}} NodeInfo:{MachineID: SystemUUID: BootID: KernelVersion: OsImage: ContainerRuntimeVersion:docker:// KubeletVersion:v1.1.8-beta.0.12+7e94fe062bc7c4 KubeProxyVersion:v1.1.8-beta.0.12+7e94fe062bc7c4}}.\r\n02:31:51 I0202 10:31:51.662873   11528 nodecontroller.go:611] Nodes ReadyCondition updated. Updating timestamp: {Capacity:map[pods:{Amount:32.000 Format:DecimalSI} cpu:{Amount:0 Format:DecimalSI} memory:{Amount:0 Format:DecimalSI}] Phase: Conditions:[{Type:OutOfDisk Status:False LastHeartbeatTime:2016-02-02 10:31:43 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletHasSufficientDisk Message:kubelet has sufficient disk space available} {Type:Ready Status:True LastHeartbeatTime:2016-02-02 10:31:43 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletReady Message:kubelet is posting ready status}] Addresses:[{Type:LegacyHostIP Address:172.17.0.3} {Type:InternalIP Address:172.17.0.3}] DaemonEndpoints:{KubeletEndpoint:{Port:10250}} NodeInfo:{MachineID: SystemUUID: BootID: KernelVersion: OsImage: ContainerRuntimeVersion:docker:// KubeletVersion:v1.1.8-beta.0.12+7e94fe062bc7c4 KubeProxyVersion:v1.1.8-beta.0.12+7e94fe062bc7c4}}\r\n02:31:51  vs {Capacity:map[pods:{Amount:32.000 Format:DecimalSI} cpu:{Amount:0 Format:DecimalSI} memory:{Amount:0 Format:DecimalSI}] Phase: Conditions:[{Type:OutOfDisk Status:False LastHeartbeatTime:2016-02-02 10:31:49 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletHasSufficientDisk Message:kubelet has sufficient disk space available} {Type:Ready Status:True LastHeartbeatTime:2016-02-02 10:31:49 +0000 UTC LastTransitionTime:2016-02-02 10:26:41 +0000 UTC Reason:KubeletReady Message:kubelet is posting ready status}] Addresses:[{Type:LegacyHostIP Address:172.17.0.3} {Type:InternalIP Address:172.17.0.3}] DaemonEndpoints:{KubeletEndpoint:{Port:10250}} NodeInfo:{MachineID: SystemUUID: BootID: KernelVersion: OsImage: ContainerRuntimeVersion:docker:// KubeletVersion:v1.1.8-beta.0.12+7e94fe062bc7c4 KubeProxyVersion:v1.1.8-beta.0.12+7e94fe062bc7c4}}.\r\n02:31:52 I0202 10:31:52.081478   11528 config.go:254] Setting pods for source file\r\n02:31:52 I0202 10:31:52.785289   11528 util.go:508] Default route transits interface \"eth0\"\r\n02:31:52 I0202 10:31:52.785440   11528 util.go:353] Interface eth0 is up\r\n02:31:52 I0202 10:31:52.785507   11528 util.go:398] Interface \"eth0\" has 2 addresses :[172.17.0.3/16 fe80::42:acff:fe11:3/64].\r\n02:31:52 I0202 10:31:52.785534   11528 util.go:365] Checking addr  172.17.0.3/16.\r\n02:31:52 I0202 10:31:52.785545   11528 util.go:374] IP found 172.17.0.3\r\n02:31:52 I0202 10:31:52.785555   11528 util.go:404] valid IPv4 address for interface \"eth0\" found as 172.17.0.3.\r\n02:31:52 I0202 10:31:52.785565   11528 util.go:514] Choosing IP 172.17.0.3 \r\n02:31:53 I0202 10:31:53.081807   11528 config.go:254] Setting pods for source file\r\n02:31:54 I0202 10:31:54.082115   11528 config.go:254] Setting pods for source file\r\n02:31:54 F0202 10:31:54.673137   11528 integration.go:846] FAILED: unexpected endpoints: timed out waiting for the condition\r\n02:31:54 !!! Error in ./hack/test-integration.sh:57\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/180042223","html_url":"https://github.com/kubernetes/kubernetes/issues/20500#issuecomment-180042223","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20500","id":180042223,"user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"created_at":"2016-02-04T20:45:52Z","updated_at":"2016-02-04T20:45:52Z","body":"I'm more inclined to believe that endpoints are awol because of either a watch cache bug or something to do with the recent label selector refactor (not sure if this is in yet), because the endpoints controller is fairly simple and hasn't been modified in a while. If this is happening, it would explain some of the services flake too."}}