{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/4641","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/4641/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/4641/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/4641/events","html_url":"https://github.com/coreos/etcd/issues/4641","id":137095149,"number":4641,"title":"functional-tester: WaitHealth fails to connect to member","user":{"login":"gyuho","id":6799218,"avatar_url":"https://avatars.githubusercontent.com/u/6799218?v=3","gravatar_id":"","url":"https://api.github.com/users/gyuho","html_url":"https://github.com/gyuho","followers_url":"https://api.github.com/users/gyuho/followers","following_url":"https://api.github.com/users/gyuho/following{/other_user}","gists_url":"https://api.github.com/users/gyuho/gists{/gist_id}","starred_url":"https://api.github.com/users/gyuho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gyuho/subscriptions","organizations_url":"https://api.github.com/users/gyuho/orgs","repos_url":"https://api.github.com/users/gyuho/repos","events_url":"https://api.github.com/users/gyuho/events{/privacy}","received_events_url":"https://api.github.com/users/gyuho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/etcd/labels/area/testing","name":"area/testing","color":"bfe5bf"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-28T20:54:53Z","updated_at":"2016-02-28T22:46:34Z","closed_at":null,"body":"\r\n```\r\n2016-02-28 14:35:29.551551 I | etcd-tester: [round#205 case#5] start injecting failure...\r\n2016-02-28 14:35:29.553905 I | etcd-tester: Stopping 10.240.0.4:2378\r\n2016-02-28 14:35:33.614126 I | etcd-tester: [round#205 case#5] injected failure\r\n2016-02-28 14:35:33.614167 I | etcd-tester: [round#205 case#5] start recovering failure...\r\n2016-02-28 14:35:34.655659 I | etcd-tester: [IMPORTANT] #0: WaitHealth error [2] rpc error: code = 4 desc = \"context deadline exceeded\" (10.240.0.4:2378) SHUTDOWN <nil> (took 1.00013115s)\r\n2016-02-28 14:35:36.699561 I | etcd-tester: [IMPORTANT] #1: WaitHealth error [2] rpc error: code = 4 desc = \"context deadline exceeded\" (10.240.0.4:2378) SHUTDOWN <nil> (took 1.000194454s)\r\n2016-02-28 14:35:38.739599 I | etcd-tester: [IMPORTANT] #2: WaitHealth error [2] rpc error: code = 4 desc = \"context deadline exceeded\" (10.240.0.4:2378) SHUTDOWN <nil> (took 1.000277694s)\r\n\r\n2016-02-28 14:49:44.188115 I | etcd-tester: [IMPORTANT] #298: WaitHealth error [2] rpc error: code = 2 desc = \"grpc: the client connection is closing\" (10.240.0.4:2378) SHUTDOWN <nil> (took 2.587471347s)\r\n2016-02-28 14:49:48.236009 I | etcd-tester: [IMPORTANT] #299: WaitHealth error [2] rpc error: code = 4 desc = \"context deadline exceeded\" (10.240.0.4:2378) SHUTDOWN <nil> (took 3.002097277s)\r\n2016-02-28 14:49:49.236179 I | etcd-tester: [round#205 case#5] recovery error: [2] rpc error: code = 4 desc = \"context deadline exceeded\" (10.240.0.4:2378) SHUTDOWN <nil> (took 3.002097277s)\r\n2016-02-28 14:49:49.237324 I | etcd-tester: [round#205 case#5] cleaning up...\r\n```\r\n\r\nWith:\r\n\r\n```go\r\nfunc (c *cluster) WaitHealth() error {\r\n\tvar err error\r\n\t// wait 60s to check cluster health.\r\n\t// TODO: set it to a reasonable value. It is set that high because\r\n\t// follower may use long time to catch up the leader when reboot under\r\n\t// reasonable workload (https://github.com/coreos/etcd/issues/2698)\r\n\thealthFunc, urls := setHealthKey, c.GRPCURLs\r\n\tif c.v2Only {\r\n\t\thealthFunc, urls = setHealthKeyV2, c.ClientURLs\r\n\t}\r\n\tfor i := 0; i < 300; i++ {\r\n\t\terr = healthFunc(urls)\r\n\t\tif err == nil {\r\n\t\t\treturn nil\r\n\t\t} else {\r\n\t\t\tplog.Printf(\"[IMPORTANT] #%d: WaitHealth error %v\", i, err)\r\n\t\t}\r\n\t\tif i > 180 {\r\n\t\t\tsetHealthKeyTimeout = 3 * time.Second\t\r\n\t\t}\r\n\t\ttime.Sleep(time.Second)\r\n\t}\r\n\treturn err\r\n}\r\n\r\nvar setHealthKeyTimeout = time.Second\r\n\r\n// setHealthKey sets health key on all given urls.\r\nfunc setHealthKey(us []string) error {\r\n\tfor _, u := range us {\r\n\t\tconn, err := grpc.Dial(u, grpc.WithInsecure(), grpc.WithTimeout(5*time.Second))\r\n\t\tif err != nil {\r\n\t\t\ts, e := conn.State()\r\n\t\t\treturn fmt.Errorf(\"[1] %v (%s) %v %v\", err, u, s, e)\r\n\t\t}\r\n\t\tctx, cancel := context.WithTimeout(context.Background(), setHealthKeyTimeout)\r\n\t\tkvc := pb.NewKVClient(conn)\r\n\t\tst := time.Now()\r\n\t\t_, err = kvc.Put(ctx, &pb.PutRequest{Key: []byte(\"health\"), Value: []byte(\"good\")})\r\n\t\tcancel()\r\n\t\tconn.Close()\r\n\t\tif err != nil {\r\n\t\t\ts, e := conn.State()\r\n\t\t\treturn fmt.Errorf(\"[2] %v (%s) %v %v (took %v)\", err, u, s, e, time.Since(st))\r\n\t\t}\r\n\t}\r\n\treturn nil\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/189962196","html_url":"https://github.com/coreos/etcd/issues/4641#issuecomment-189962196","issue_url":"https://api.github.com/repos/coreos/etcd/issues/4641","id":189962196,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2016-02-28T22:46:34Z","updated_at":"2016-02-28T22:46:34Z","body":"@gyuho Can you check the log to see what happened?"}}