{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/3779","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/3779/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/3779/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/3779/events","html_url":"https://github.com/coreos/etcd/issues/3779","id":114155678,"number":3779,"title":"Rapid leader elections and request timeouts causing write failures","user":{"login":"jenspinney","id":3200507,"avatar_url":"https://avatars.githubusercontent.com/u/3200507?v=3","gravatar_id":"","url":"https://api.github.com/users/jenspinney","html_url":"https://github.com/jenspinney","followers_url":"https://api.github.com/users/jenspinney/followers","following_url":"https://api.github.com/users/jenspinney/following{/other_user}","gists_url":"https://api.github.com/users/jenspinney/gists{/gist_id}","starred_url":"https://api.github.com/users/jenspinney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jenspinney/subscriptions","organizations_url":"https://api.github.com/users/jenspinney/orgs","repos_url":"https://api.github.com/users/jenspinney/repos","events_url":"https://api.github.com/users/jenspinney/events{/privacy}","received_events_url":"https://api.github.com/users/jenspinney/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/etcd/labels/area/performance","name":"area/performance","color":"bfe5bf"},{"url":"https://api.github.com/repos/coreos/etcd/labels/kind/question","name":"kind/question","color":"c7def8"},{"url":"https://api.github.com/repos/coreos/etcd/labels/kind/support","name":"kind/support","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"gyuho","id":6799218,"avatar_url":"https://avatars.githubusercontent.com/u/6799218?v=3","gravatar_id":"","url":"https://api.github.com/users/gyuho","html_url":"https://github.com/gyuho","followers_url":"https://api.github.com/users/gyuho/followers","following_url":"https://api.github.com/users/gyuho/following{/other_user}","gists_url":"https://api.github.com/users/gyuho/gists{/gist_id}","starred_url":"https://api.github.com/users/gyuho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gyuho/subscriptions","organizations_url":"https://api.github.com/users/gyuho/orgs","repos_url":"https://api.github.com/users/gyuho/repos","events_url":"https://api.github.com/users/gyuho/events{/privacy}","received_events_url":"https://api.github.com/users/gyuho/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2015-10-29T21:53:19Z","updated_at":"2016-04-07T22:35:27Z","closed_at":null,"body":"Hello,\r\n\r\nWhile running ETCD in the Diego project of Cloud Foundry, we've noticed problems when we insert many records into the store. We've seen the ETCD nodes apparently failing to communicate with each other and rapidly increasing the raft term. During these time periods, some writes fail.\r\n\r\nIn the logs, we see many messages like:\r\n\r\n```\r\n2015/10/29 18:56:40 etcdhttp: got unexpected response error (etcdserver: request timed out)\r\n```\r\n\r\nWe've created a small executable that reproduces the issues we're seeing on our environment, and have created a repo with that source code and a fuller description of our environment: https://github.com/jenspinney/etcdstress\r\n\r\nWould you be able to take a look and let us know if this repros in your environment? This is causing us a lot of problems at the moment when we increase the amount of data we store in ETCD.\r\n\r\nLet us know if you need other information. Thank you!\r\n\r\n@jenspinney & @andrew-edgar"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/207121587","html_url":"https://github.com/coreos/etcd/issues/3779#issuecomment-207121587","issue_url":"https://api.github.com/repos/coreos/etcd/issues/3779","id":207121587,"user":{"login":"gyuho","id":6799218,"avatar_url":"https://avatars.githubusercontent.com/u/6799218?v=3","gravatar_id":"","url":"https://api.github.com/users/gyuho","html_url":"https://github.com/gyuho","followers_url":"https://api.github.com/users/gyuho/followers","following_url":"https://api.github.com/users/gyuho/following{/other_user}","gists_url":"https://api.github.com/users/gyuho/gists{/gist_id}","starred_url":"https://api.github.com/users/gyuho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gyuho/subscriptions","organizations_url":"https://api.github.com/users/gyuho/orgs","repos_url":"https://api.github.com/users/gyuho/repos","events_url":"https://api.github.com/users/gyuho/events{/privacy}","received_events_url":"https://api.github.com/users/gyuho/received_events","type":"User","site_admin":false},"created_at":"2016-04-07T22:35:27Z","updated_at":"2016-04-07T22:35:27Z","body":"@jenspinney Kindly ping. Sorry to get back to this late. I am trying to take another look\r\nat this issue. Are you still seeing this error? Was there any network partition or any kind\r\nof network issue that might have made one node lose its leader (as seen in your log)?\r\n\r\nI have seen similar case in recent functional-tests https://github.com/coreos/etcd/issues/4855#issuecomment-200970476, but it's resolved. We are pretty confident now that this won't happen in v3, of which alpha release is coming\r\nin a few weeks.\r\n\r\nIf you can provide simpler way to reproduce this issue, it would be very helpful. Thanks!\r\n"}}