{"action":"created","issue":{"url":"https://api.github.com/repos/shazow/urllib3/issues/805","repository_url":"https://api.github.com/repos/shazow/urllib3","labels_url":"https://api.github.com/repos/shazow/urllib3/issues/805/labels{/name}","comments_url":"https://api.github.com/repos/shazow/urllib3/issues/805/comments","events_url":"https://api.github.com/repos/shazow/urllib3/issues/805/events","html_url":"https://github.com/shazow/urllib3/issues/805","id":136020481,"number":805,"title":"Connection pool queue can still be exhausted by gevent timeouts","user":{"login":"ml31415","id":2386612,"avatar_url":"https://avatars.githubusercontent.com/u/2386612?v=3","gravatar_id":"","url":"https://api.github.com/users/ml31415","html_url":"https://github.com/ml31415","followers_url":"https://api.github.com/users/ml31415/followers","following_url":"https://api.github.com/users/ml31415/following{/other_user}","gists_url":"https://api.github.com/users/ml31415/gists{/gist_id}","starred_url":"https://api.github.com/users/ml31415/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ml31415/subscriptions","organizations_url":"https://api.github.com/users/ml31415/orgs","repos_url":"https://api.github.com/users/ml31415/repos","events_url":"https://api.github.com/users/ml31415/events{/privacy}","received_events_url":"https://api.github.com/users/ml31415/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-02-24T10:35:22Z","updated_at":"2016-02-26T00:47:01Z","closed_at":null,"body":"A similar error was discussed here https://github.com/shazow/urllib3/issues/644 already and some fix applied, but the problem still partly exists. Consider this code:\r\n\r\n```python\r\nfrom gevent import monkey, spawn, sleep, Timeout, joinall\r\nmonkey.patch_all()\r\n\r\nimport sys\r\nfrom itertools import product\r\nfrom urllib3.connectionpool import HTTPConnectionPool\r\n\r\ndef timed_read(cp, i, timeout, release=None, preload_content=True):\r\n    try:\r\n        with Timeout(timeout):\r\n            resp = cp.urlopen('GET', 'http://httpbin.org/redirect/:1',\r\n                              release_conn=release,\r\n                              preload_content=preload)\r\n            print \"%s opened\" % i\r\n            sys.stdout.flush()\r\n            data = resp.data\r\n    except Timeout:\r\n        print \"%s timed out\" % i\r\n    else:\r\n        content_len = int(resp.headers['content-length'])\r\n        if len(data) != content_len:\r\n            print \"%s incomplete read: %s/%s\" % (i, len(data), content_len)\r\n        else:\r\n            print \"%s finished\" % i\r\n    sys.stdout.flush()\r\n\r\n\r\ndef main(timeout=10, release=None, preload=True, poolsize=2):\r\n    print '-' * 10 + \"  timeout=%s preload=%s release=%s \" % (timeout, preload, release) + '-' * 10\r\n    cp = HTTPConnectionPool('httpbin.org', block=True, maxsize=poolsize, timeout=None)\r\n    group = []\r\n    for i in range(3):\r\n        group.append(spawn(timed_read, cp, i, timeout, release=release, preload_content=preload))\r\n    joinall(group)\r\n\r\n    current_size = cp.pool.qsize()\r\n    if current_size != poolsize:\r\n        print \"poolsize decreased: %s\" % current_size\r\n\r\n\r\nfor timeout, preload in product([10, 4, 0.1], [True, False]):\r\n    main(timeout=timeout, preload=preload)\r\n```\r\n\r\nSetting a timeout that way may seem stupid, as it could be set on the connection itself, but it's just an example for having a timeout running on a longer running task including several calls into the connection pool. What happens is, that the connections are not properly returned back into the queue, and the queue finally runs empty, and any further call to it blocks forever:\r\n\r\n<pre>\r\n----------  timeout=10 preload=True release=None ----------\r\n1 opened\r\n1 finished\r\n0 opened\r\n0 finished\r\n2 opened\r\n2 finished\r\n----------  timeout=10 preload=False release=None ----------\r\n0 opened\r\n0 finished\r\n1 opened\r\n1 finished\r\n2 opened\r\n2 finished\r\n----------  timeout=4 preload=True release=None ----------\r\n1 opened\r\n1 finished\r\n0 timed out\r\n2 timed out\r\n----------  timeout=4 preload=False release=None ----------\r\n1 opened\r\n1 finished\r\n0 opened\r\n0 finished\r\n2 timed out\r\npoolsize decreased: 1\r\n----------  timeout=0.1 preload=True release=None ----------\r\n0 timed out\r\n1 timed out\r\n2 timed out\r\n----------  timeout=0.1 preload=False release=None ----------\r\n0 timed out\r\n1 timed out\r\n2 timed out\r\npoolsize decreased: 0\r\n</pre>\r\n\r\nSimply setting `preload_content` to `True` is not an option, as e.g. `False` is set by default by requests.\r\n\r\nThe main problem is in https://github.com/shazow/urllib3/blob/master/urllib3/response.py#L246 . The `Timeout` does not derive from `Exception`, so `self._original_response` is not closed in the except clause, and therefore the connection is not released in the finally clause.\r\n\r\n<bountysource-plugin>\r\n---\r\nWant to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/31218594-connection-pool-queue-can-still-be-exhausted-by-gevent-timeouts?utm_campaign=plugin&utm_content=tracker%2F192525&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F192525&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"},"comment":{"url":"https://api.github.com/repos/shazow/urllib3/issues/comments/189054743","html_url":"https://github.com/shazow/urllib3/issues/805#issuecomment-189054743","issue_url":"https://api.github.com/repos/shazow/urllib3/issues/805","id":189054743,"user":{"login":"ml31415","id":2386612,"avatar_url":"https://avatars.githubusercontent.com/u/2386612?v=3","gravatar_id":"","url":"https://api.github.com/users/ml31415","html_url":"https://github.com/ml31415","followers_url":"https://api.github.com/users/ml31415/followers","following_url":"https://api.github.com/users/ml31415/following{/other_user}","gists_url":"https://api.github.com/users/ml31415/gists{/gist_id}","starred_url":"https://api.github.com/users/ml31415/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ml31415/subscriptions","organizations_url":"https://api.github.com/users/ml31415/orgs","repos_url":"https://api.github.com/users/ml31415/repos","events_url":"https://api.github.com/users/ml31415/events{/privacy}","received_events_url":"https://api.github.com/users/ml31415/received_events","type":"User","site_admin":false},"created_at":"2016-02-26T00:47:01Z","updated_at":"2016-02-26T00:47:01Z","body":"Though I think the changes in #805 are the right thing to do."}}