{"action":"opened","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1780","repository_url":"https://api.github.com/repos/hashicorp/consul","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1780/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1780/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1780/events","html_url":"https://github.com/hashicorp/consul/issues/1780","id":137832154,"number":1780,"title":"stale request intermittently fails during leader election","user":{"login":"dropje86","id":6005868,"avatar_url":"https://avatars.githubusercontent.com/u/6005868?v=3","gravatar_id":"","url":"https://api.github.com/users/dropje86","html_url":"https://github.com/dropje86","followers_url":"https://api.github.com/users/dropje86/followers","following_url":"https://api.github.com/users/dropje86/following{/other_user}","gists_url":"https://api.github.com/users/dropje86/gists{/gist_id}","starred_url":"https://api.github.com/users/dropje86/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dropje86/subscriptions","organizations_url":"https://api.github.com/users/dropje86/orgs","repos_url":"https://api.github.com/users/dropje86/repos","events_url":"https://api.github.com/users/dropje86/events{/privacy}","received_events_url":"https://api.github.com/users/dropje86/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-02T10:17:17Z","updated_at":"2016-03-02T10:17:17Z","closed_at":null,"body":"During a failure or node leave of the leader we sometimes experience hickups in requests, even though we query **with** the stale parameter set.\r\n\r\nreproduce:\r\n\r\n```python\r\n#!/usr/bin/env python\r\n\r\nimport requests\r\nimport time\r\n\r\nleader = \"\"\r\n\r\nwith requests.Session() as session:\r\n    while True:\r\n        try:\r\n            r = session.get('http://localhost:8500/v1/health/service/<service>?tag=<tag>&passing&stale')\r\n            if r.status_code is not 200:\r\n                print(\"Request failed: {}\".format(r.status_code))\r\n                print(\"Response: {}\".format(r.content))\r\n            l = session.get('http://localhost:8500/v1/status/leader')\r\n            if leader == l.content:\r\n                continue\r\n            leader = l.content\r\n            print(\"leader: {}\".format(leader))\r\n            time.sleep(0.01)\r\n        except KeyboardInterrupt:\r\n            exit()\r\n```\r\n\r\nEvery now and then it seems the connection is aborted or the client still tries to reconnect to the failed node (getting a connection refused). We're trying to work around this by letting the application retry on failure but imho it should be the agent's responsibility to query a different node when using stale lookups.\r\n\r\nHere's output tested with stopping the active leader:\r\n\r\n```shell\r\nleader: \"10.40.31.164:8300\"\r\nRequest failed: 500\r\nResponse: rpc error: EOF\r\nleader: \"\"\r\nleader: \"10.40.31.166:8300\"\r\nleader: \"\"\r\nRequest failed: 500\r\nResponse: rpc error: stream closed\r\nleader: rpc error: stream closed\r\nRequest failed: 500\r\nResponse: rpc error: failed to get conn: dial tcp 10.40.31.166:8300: getsockopt: connection refused\r\nleader: rpc error: failed to get conn: dial tcp 10.40.31.166:8300: getsockopt: connection refused\r\nleader: \"10.40.31.166:8300\"\r\nleader: \"\"\r\nleader: \"10.40.31.165:8300\"\r\n```"}}