{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/6685","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/6685/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/6685/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/6685/events","html_url":"https://github.com/coreos/etcd/issues/6685","id":184027840,"number":6685,"title":"functional-tester: inconsistent hash error with lease stresser + 5 node","user":{"login":"gyuho","id":6799218,"avatar_url":"https://avatars.githubusercontent.com/u/6799218?v=3","gravatar_id":"","url":"https://api.github.com/users/gyuho","html_url":"https://github.com/gyuho","followers_url":"https://api.github.com/users/gyuho/followers","following_url":"https://api.github.com/users/gyuho/following{/other_user}","gists_url":"https://api.github.com/users/gyuho/gists{/gist_id}","starred_url":"https://api.github.com/users/gyuho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gyuho/subscriptions","organizations_url":"https://api.github.com/users/gyuho/orgs","repos_url":"https://api.github.com/users/gyuho/repos","events_url":"https://api.github.com/users/gyuho/events{/privacy}","received_events_url":"https://api.github.com/users/gyuho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/etcd/labels/component/functional-tester","name":"component/functional-tester","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-10-19T17:39:10Z","updated_at":"2016-10-26T17:09:12Z","closed_at":null,"body":"@fanminshi \r\n\r\nGo to line 155276 in the tester.log\r\n\r\nhttps://storage.googleapis.com/etcd-functional-tester/lease-hash-20161019/tester.log\r\n\r\n```\r\n2016-10-18 23:26:48.719298 I | etcd-tester: alive leases map[9011799448683162664:2016-10-18 23:26:46.041491347 +0000 UTC 9011799448683162386:2016-10-18 23:26:45.932348378 +0000 UTC]\r\n2016-10-18 23:26:48.764569 I | etcd-tester: alive leases map[8084620875400444400:2016-10-18 23:26:46.061914951 +0000 UTC 8084620875400444456:2016-10-18 23:26:46.093931904 +0000 UTC 8084620875400444404:2016-10-18 23:26:46.061926252 +0000 UTC 8084620875400444396:2016-10-18 23:26:46.062535836 +0000 UTC]\r\n2016-10-18 23:26:48.773281 I | etcd-tester: alive leases map[3235651451604947091:2016-10-18 23:26:45.979305112 +0000 UTC]\r\n2016-10-18 23:26:48.858784 I | etcd-tester: alive leases map[4991210881352861129:2016-10-18 23:26:46.108168558 +0000 UTC 4991210881352860892:2016-10-18 23:26:46.016549526 +0000 UTC 4991210881352860890:2016-10-18 23:26:46.016642209 +0000 UTC 4991210881352861068:2016-10-18 23:26:46.075974333 +0000 UTC 4991210881352860994:2016-10-18 23:26:46.046630186 +0000 UTC]\r\n2016-10-18 23:26:48.875467 I | etcd-tester: alive leases map[8887668983953728954:2016-10-18 23:26:45.86580866 +0000 UTC 8887668983953729274:2016-10-18 23:26:45.944533881 +0000 UTC 8887668983953729496:2016-10-18 23:26:46.114056772 +0000 UTC 8887668983953729333:2016-10-18 23:26:46.012304546 +0000 UTC 8887668983953729498:2016-10-18 23:26:46.114183138 +0000 UTC]\r\n2016-10-18 23:26:49.497262 I | etcd-tester: [round#33 case#8] updated current revision to 17917943\r\n2016-10-18 23:26:49.497301 I | etcd-tester: [round#33 case#8] starting the stressers...\r\n2016-10-18 23:26:49.497451 I | etcd-tester: stresser \"10.240.0.6:2379\" is started\r\n2016-10-18 23:26:49.497464 I | etcd-tester: lease Stresser 10.240.0.6:2379 starting ...\r\n2016-10-18 23:26:49.497627 I | etcd-tester: stresser \"10.240.0.13:2379\" is started\r\n2016-10-18 23:26:49.497638 I | etcd-tester: lease Stresser 10.240.0.13:2379 starting ...\r\n2016-10-18 23:26:49.497749 I | etcd-tester: stresser \"10.240.0.14:2379\" is started\r\n2016-10-18 23:26:49.497757 I | etcd-tester: lease Stresser 10.240.0.14:2379 starting ...\r\n2016-10-18 23:26:49.497911 I | etcd-tester: stresser \"10.240.0.15:2379\" is started\r\n2016-10-18 23:26:49.497923 I | etcd-tester: lease Stresser 10.240.0.15:2379 starting ...\r\n2016-10-18 23:26:49.498050 I | etcd-tester: stresser \"10.240.0.16:2379\" is started\r\n2016-10-18 23:26:49.498059 I | etcd-tester: lease Stresser 10.240.0.16:2379 starting ...\r\n2016-10-18 23:26:49.498068 I | etcd-tester: [round#33 case#8] started stressers\r\n2016-10-18 23:26:49.498075 I | etcd-tester: [round#33 case#8] succeed!\r\n2016-10-18 23:26:49.599031 I | etcd-tester: [round#33 case#9] injecting failure \"slow down leader's network by adding 500 ms latency\"\r\n2016-10-18 23:26:49.668761 I | etcd-tester: [round#33 case#9] injected failure\r\n2016-10-18 23:26:49.668798 I | etcd-tester: [round#33 case#9] recovering failure \"slow down leader's network by adding 500 ms latency\"\r\n2016-10-18 23:26:50.243108 E | etcd-tester: lease creation error: (rpc error: code = 2 desc = etcdserver: too many requests)\r\n2016-10-18 23:26:50.243440 E | etcd-tester: lease creation error: (rpc error: code = 2 desc = etcdserver: too many requests)\r\n2016-10-18 23:26:50.243469 E | etcd-tester: lease creation error: (rpc error: code = 2 desc = etcdserver: too many requests)\r\n2016-10-18 23:26:50.243486 E | etcd-tester: lease creation error: (rpc error: code = 2 desc = etcdserver: too many requests)\r\n2016-10-18 23:26:54.750568 I | etcd-tester: [round#33 case#9] recovered failure\r\n2016-10-18 23:26:54.750607 I | etcd-tester: [round#33 case#9] canceling the stressers...\r\n2016-10-18 23:26:54.751067 I | etcd-tester: stresser \"10.240.0.6:2379\" is canceled\r\n2016-10-18 23:26:54.755811 I | etcd-tester: lease stresser \"10.240.0.6:2379\" is canceled\r\n2016-10-18 23:26:54.756330 I | etcd-tester: stresser \"10.240.0.13:2379\" is canceled\r\n2016-10-18 23:26:54.757231 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.757262 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.764764 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.764803 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.764820 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.764845 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.764894 I | etcd-tester: lease stresser \"10.240.0.13:2379\" is canceled\r\n2016-10-18 23:26:54.765121 I | etcd-tester: stresser \"10.240.0.14:2379\" is canceled\r\n2016-10-18 23:26:54.765644 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.765664 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.765680 E | etcd-tester: lease creation error: (rpc error: code = 1 desc = context canceled)\r\n2016-10-18 23:26:54.768938 I | etcd-tester: lease stresser \"10.240.0.14:2379\" is canceled\r\n2016-10-18 23:26:54.769591 I | etcd-tester: stresser \"10.240.0.15:2379\" is canceled\r\n2016-10-18 23:26:54.773441 I | etcd-tester: lease stresser \"10.240.0.15:2379\" is canceled\r\n2016-10-18 23:26:54.773708 I | etcd-tester: stresser \"10.240.0.16:2379\" is canceled\r\n2016-10-18 23:26:54.774284 I | etcd-tester: lease stresser \"10.240.0.16:2379\" is canceled\r\n2016-10-18 23:26:54.774299 I | etcd-tester: [round#33 case#9] canceled stressers\r\n2016-10-18 23:26:54.774308 I | etcd-tester: fetching current revisions...\r\n2016-10-18 23:26:56.531165 I | etcd-tester: all members are consistent with current revisions [revisions: map[http://10.240.0.13:2379:17973155 http://10.240.0.14:2379:17973155 http://10.240.0.15:2379:17973155 http://10.240.0.16:2379:17973155 http://10.240.0.6:2379:17973155]]\r\n2016-10-18 23:26:56.531219 I | etcd-tester: checking current storage hashes...\r\n2016-10-18 23:26:56.531241 I | etcd-tester: [round#33 case#9] inconsistent hashes [map[http://10.240.0.14:2379:1601651870 http://10.240.0.15:2379:1993083154 http://10.240.0.16:2379:1993083154 http://10.240.0.6:2379:1601651870 http://10.240.0.13:2379:1601651870]]\r\n2016-10-18 23:26:56.531252 W | etcd-tester: [round#33 case#9] functional-tester returning with tt.checkConsistency error (inconsistent hashes [map[http://10.240.0.14:2379:1601651870 http://10.240.0.15:2379:1993083154 http://10.240.0.16:2379:1993083154 http://10.240.0.6:2379:1601651870 http://10.240.0.13:2379:1601651870]])\r\n2016-10-18 23:26:56.531282 I | etcd-tester: [round#33 case#9] cleaning up...\r\n```\r\n\r\n```\r\nhttp://10.240.0.6:2379:1601651870\r\nhttp://10.240.0.13:2379:1601651870\r\nhttp://10.240.0.14:2379:1601651870\r\nhttp://10.240.0.15:2379:1993083154\r\nhttp://10.240.0.16:2379:1993083154\r\n```\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/256414840","html_url":"https://github.com/coreos/etcd/issues/6685#issuecomment-256414840","issue_url":"https://api.github.com/repos/coreos/etcd/issues/6685","id":256414840,"user":{"login":"fanminshi","id":4983345,"avatar_url":"https://avatars.githubusercontent.com/u/4983345?v=3","gravatar_id":"","url":"https://api.github.com/users/fanminshi","html_url":"https://github.com/fanminshi","followers_url":"https://api.github.com/users/fanminshi/followers","following_url":"https://api.github.com/users/fanminshi/following{/other_user}","gists_url":"https://api.github.com/users/fanminshi/gists{/gist_id}","starred_url":"https://api.github.com/users/fanminshi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fanminshi/subscriptions","organizations_url":"https://api.github.com/users/fanminshi/orgs","repos_url":"https://api.github.com/users/fanminshi/repos","events_url":"https://api.github.com/users/fanminshi/events{/privacy}","received_events_url":"https://api.github.com/users/fanminshi/received_events","type":"User","site_admin":false},"created_at":"2016-10-26T17:09:12Z","updated_at":"2016-10-26T17:09:12Z","body":"Lease expired during the time that hashes not being stable for about 20 seconds.  Therefore, invariant check failed to find the lease. Increasing lease TTL to 60 or 120 seconds will solve this problem.\r\n```\r\n2016-10-26 04:08:07.875563 I | etcd-tester: [round#33 case#43] canceled stressers\r\n2016-10-26 04:08:08.372278 I | etcd-tester: current revisions are not consistent: revisions [revisions: map[http://10.240.0.27:2379:14708495 http://10.240.0.28:2379:14708500 http://10.240.0.29:2379:14708500]]\r\n2016-10-26 04:08:12.358329 I | etcd-tester: revisions are stable: revisions [revisions: map[http://10.240.0.27:2379:14708500 http://10.240.0.28:2379:14708500 http://10.240.0.29:2379:14708500]]\r\n2016-10-26 04:08:16.338588 I | etcd-tester: hashes are not stable: [current hashes: map[http://10.240.0.27:2379:1541071615 http://10.240.0.28:2379:1541071615 http://10.240.0.29:2379:1541071615]] [previous hashes: map[http://10.240.0.27:2379:2907886245 http://10.240.0.28:2379:2907886245 http://10.240.0.29:2379:2907886245]]\r\n2016-10-26 04:08:20.313809 I | etcd-tester: revisions are stable: revisions [revisions: map[http://10.240.0.27:2379:14708502 http://10.240.0.28:2379:14708502 http://10.240.0.29:2379:14708502]]\r\n2016-10-26 04:08:24.329283 I | etcd-tester: hashes are not stable: [current hashes: map[http://10.240.0.27:2379:1061930828 http://10.240.0.28:2379:1061930828 http://10.240.0.29:2379:1061930828]] [previous hashes: map[http://10.240.0.27:2379:1541071615 http://10.240.0.28:2379:1541071615 http://10.240.0.29:2379:1541071615]]\r\n2016-10-26 04:08:28.291078 I | etcd-tester: revisions are stable: revisions [revisions: map[http://10.240.0.28:2379:14708503 http://10.240.0.29:2379:14708503 http://10.240.0.27:2379:14708503]]\r\n2016-10-26 04:08:32.250377 I | etcd-tester: hashes are stable: hashes [hashes: map[http://10.240.0.27:2379:1061930828 http://10.240.0.28:2379:1061930828 http://10.240.0.29:2379:1061930828]]\r\n2016-10-26 04:08:32.310934 I | etcd-tester: [round#33 case#43] lease stresser encounters error: (lease 8747213129729321485 expected expired=false, got true)\r\n```"}}
{"actor":{"display_login":"fanminshi"}}