{"action":"created","comment":{"url":"https://api.github.com/repos/shazow/urllib3/pulls/comments/49436781","id":49436781,"diff_hunk":"@@ -162,6 +162,34 @@ def connect(self):\n         conn = self._new_conn()\n         self._prepare_conn(conn)\n \n+    def request_chunked(self, method, url, body=None, headers={}):\n+        \"\"\"\n+        Alternative to the common request method, which sends the\n+        body with chunked encoding and not as one block\n+        \"\"\"\n+        header_names = set(k.lower() for k in headers)\n+        self.putrequest(method, url, skip_accept_encoding=True)\n+        for header, value in headers.items():\n+            self.putheader(header, value)\n+        if 'transfer-encoding' not in header_names:\n+            self.putheader('Transfer-Encoding', 'chunked')\n+        self.endheaders()\n+\n+        if body is not None:\n+            if isinstance(body, six.string_types):\n+                body = [body]\n+            for chunk in body:\n+                if not chunk:\n+                    continue\n+                len_str = hex(len(chunk))[2:]\n+                self.send(len_str.encode('utf-8'))\n+                self.send(b'\\r\\n')\n+                self.send(chunk.encode('utf-8'))\n+                self.send(b'\\r\\n')","path":"urllib3/connection.py","position":27,"original_position":27,"commit_id":"fec75a28b27caaaf6bb546fa9eca9ec8cf2575b6","original_commit_id":"fec75a28b27caaaf6bb546fa9eca9ec8cf2575b6","user":{"login":"Lukasa","id":1382556,"avatar_url":"https://avatars.githubusercontent.com/u/1382556?v=3","gravatar_id":"","url":"https://api.github.com/users/Lukasa","html_url":"https://github.com/Lukasa","followers_url":"https://api.github.com/users/Lukasa/followers","following_url":"https://api.github.com/users/Lukasa/following{/other_user}","gists_url":"https://api.github.com/users/Lukasa/gists{/gist_id}","starred_url":"https://api.github.com/users/Lukasa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lukasa/subscriptions","organizations_url":"https://api.github.com/users/Lukasa/orgs","repos_url":"https://api.github.com/users/Lukasa/repos","events_url":"https://api.github.com/users/Lukasa/events{/privacy}","received_events_url":"https://api.github.com/users/Lukasa/received_events","type":"User","site_admin":false},"body":"Python 2.7's httplib is not the bar of acceptable that we're trying to meet, Python 3's is. The odds of a Python 2 user passing a unicode chunk here is low, but if they do it we should either error helpfully (rather than letting the implicit encode/decode catch it) or we should try to help them (I'd prefer to do the first of those things). However, the odds of a Python 3 user passing bytes here is very high, and we absolutely **must** do the right thing here in the face of that situation.\r\n\r\nTo be clear, here is the space of things a user might do and whether this function works correctly:\r\n\r\n|                            | Python 2                           | Python 3                           |\r\n|----------------------------|------------------------------------------|----------------------------------|\r\n| Bytestring: only ASCII     | Works (only on some platforms, may fail) | **Fails**: `AttributeError`        |\r\n| Bytestring: not ASCII      | **Fails**: `UnicodeDecodeError`    | **Fails**: `AttributeError`          |\r\n| Unicode string: only ASCII | Works                              | Works                              |\r\n| Unicode string: not ASCII  | **Invalid Output**: Incorrect chunk header. | **InvalidOutput**: Incorrect chunk header. |\r\n\r\nIt is not acceptable to fail in the face of a bytestring, so all of those failures need to be dealt with. It's also unacceptable to generate invalid output, because this amounts to silent data corruption, which we must not do.\r\n\r\nI strongly recommend adding tests for each of the four data types I've listed there and confirm that they all work.","created_at":"2016-01-12T10:30:04Z","updated_at":"2016-01-12T10:30:04Z","html_url":"https://github.com/shazow/urllib3/pull/775#discussion_r49436781","pull_request_url":"https://api.github.com/repos/shazow/urllib3/pulls/775","_links":{"self":{"href":"https://api.github.com/repos/shazow/urllib3/pulls/comments/49436781"},"html":{"href":"https://github.com/shazow/urllib3/pull/775#discussion_r49436781"},"pull_request":{"href":"https://api.github.com/repos/shazow/urllib3/pulls/775"}}},"pull_request":{"url":"https://api.github.com/repos/shazow/urllib3/pulls/775","id":54924082,"html_url":"https://github.com/shazow/urllib3/pull/775","diff_url":"https://github.com/shazow/urllib3/pull/775.diff","patch_url":"https://github.com/shazow/urllib3/pull/775.patch","issue_url":"https://api.github.com/repos/shazow/urllib3/issues/775","number":775,"state":"open","locked":false,"title":"WIP: Chunked transfer encoding on HTTP requests","user":{"login":"ml31415","id":2386612,"avatar_url":"https://avatars.githubusercontent.com/u/2386612?v=3","gravatar_id":"","url":"https://api.github.com/users/ml31415","html_url":"https://github.com/ml31415","followers_url":"https://api.github.com/users/ml31415/followers","following_url":"https://api.github.com/users/ml31415/following{/other_user}","gists_url":"https://api.github.com/users/ml31415/gists{/gist_id}","starred_url":"https://api.github.com/users/ml31415/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ml31415/subscriptions","organizations_url":"https://api.github.com/users/ml31415/orgs","repos_url":"https://api.github.com/users/ml31415/repos","events_url":"https://api.github.com/users/ml31415/events{/privacy}","received_events_url":"https://api.github.com/users/ml31415/received_events","type":"User","site_admin":false},"body":"I recently stumbled upon this quite hacky lines in requests: https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L380 What happens there is, that requests has to do dirty low level work, to get chunked transfer encoding going on in HTTP requests. This low level hack is fairly untested there (if I didn't miss anything obvious), and doesn't really integrate well with all the proxy stuff etc. going on on higher levels, especially error handling, as many higher levels of urllib3 are just bypassed. \r\n\r\nBut it looks like currently there was no better way to do this, as urllib3 does not support any chunked request by itself. The original `httplib.HTTPConnection` is also lacking this feature. So this pull request is basically adding some separate `request_chunked` method on HTTPConnection and exposes this functionality to upstream libraries, i.e. requests, so that it can make use of it in some later version and get rid of the misplaced lowlevel code.\r\n\r\nIn terms of implementation, it would have probably been more elgant to just add a `chunked` flag to the `HTTPConnection.request` method itself, but I thought it may be better for backwards compatibility, to have a separate function.","created_at":"2016-01-03T12:40:55Z","updated_at":"2016-01-12T10:30:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"37e23599096df19b11a6e61d577d70f4f3dbec59","assignee":{"login":"Lukasa","id":1382556,"avatar_url":"https://avatars.githubusercontent.com/u/1382556?v=3","gravatar_id":"","url":"https://api.github.com/users/Lukasa","html_url":"https://github.com/Lukasa","followers_url":"https://api.github.com/users/Lukasa/followers","following_url":"https://api.github.com/users/Lukasa/following{/other_user}","gists_url":"https://api.github.com/users/Lukasa/gists{/gist_id}","starred_url":"https://api.github.com/users/Lukasa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lukasa/subscriptions","organizations_url":"https://api.github.com/users/Lukasa/orgs","repos_url":"https://api.github.com/users/Lukasa/repos","events_url":"https://api.github.com/users/Lukasa/events{/privacy}","received_events_url":"https://api.github.com/users/Lukasa/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/shazow/urllib3/pulls/775/commits","review_comments_url":"https://api.github.com/repos/shazow/urllib3/pulls/775/comments","review_comment_url":"https://api.github.com/repos/shazow/urllib3/pulls/comments{/number}","comments_url":"https://api.github.com/repos/shazow/urllib3/issues/775/comments","statuses_url":"https://api.github.com/repos/shazow/urllib3/statuses/fec75a28b27caaaf6bb546fa9eca9ec8cf2575b6","head":{"label":"ml31415:chunked_request","ref":"chunked_request","sha":"fec75a28b27caaaf6bb546fa9eca9ec8cf2575b6","user":{"login":"ml31415","id":2386612,"avatar_url":"https://avatars.githubusercontent.com/u/2386612?v=3","gravatar_id":"","url":"https://api.github.com/users/ml31415","html_url":"https://github.com/ml31415","followers_url":"https://api.github.com/users/ml31415/followers","following_url":"https://api.github.com/users/ml31415/following{/other_user}","gists_url":"https://api.github.com/users/ml31415/gists{/gist_id}","starred_url":"https://api.github.com/users/ml31415/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ml31415/subscriptions","organizations_url":"https://api.github.com/users/ml31415/orgs","repos_url":"https://api.github.com/users/ml31415/repos","events_url":"https://api.github.com/users/ml31415/events{/privacy}","received_events_url":"https://api.github.com/users/ml31415/received_events","type":"User","site_admin":false},"repo":{"id":30180609,"name":"urllib3","full_name":"ml31415/urllib3","owner":{"login":"ml31415","id":2386612,"avatar_url":"https://avatars.githubusercontent.com/u/2386612?v=3","gravatar_id":"","url":"https://api.github.com/users/ml31415","html_url":"https://github.com/ml31415","followers_url":"https://api.github.com/users/ml31415/followers","following_url":"https://api.github.com/users/ml31415/following{/other_user}","gists_url":"https://api.github.com/users/ml31415/gists{/gist_id}","starred_url":"https://api.github.com/users/ml31415/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ml31415/subscriptions","organizations_url":"https://api.github.com/users/ml31415/orgs","repos_url":"https://api.github.com/users/ml31415/repos","events_url":"https://api.github.com/users/ml31415/events{/privacy}","received_events_url":"https://api.github.com/users/ml31415/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ml31415/urllib3","description":"Python HTTP library with thread-safe connection pooling, file post support, sanity friendly, and more.","fork":true,"url":"https://api.github.com/repos/ml31415/urllib3","forks_url":"https://api.github.com/repos/ml31415/urllib3/forks","keys_url":"https://api.github.com/repos/ml31415/urllib3/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ml31415/urllib3/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ml31415/urllib3/teams","hooks_url":"https://api.github.com/repos/ml31415/urllib3/hooks","issue_events_url":"https://api.github.com/repos/ml31415/urllib3/issues/events{/number}","events_url":"https://api.github.com/repos/ml31415/urllib3/events","assignees_url":"https://api.github.com/repos/ml31415/urllib3/assignees{/user}","branches_url":"https://api.github.com/repos/ml31415/urllib3/branches{/branch}","tags_url":"https://api.github.com/repos/ml31415/urllib3/tags","blobs_url":"https://api.github.com/repos/ml31415/urllib3/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ml31415/urllib3/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ml31415/urllib3/git/refs{/sha}","trees_url":"https://api.github.com/repos/ml31415/urllib3/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ml31415/urllib3/statuses/{sha}","languages_url":"https://api.github.com/repos/ml31415/urllib3/languages","stargazers_url":"https://api.github.com/repos/ml31415/urllib3/stargazers","contributors_url":"https://api.github.com/repos/ml31415/urllib3/contributors","subscribers_url":"https://api.github.com/repos/ml31415/urllib3/subscribers","subscription_url":"https://api.github.com/repos/ml31415/urllib3/subscription","commits_url":"https://api.github.com/repos/ml31415/urllib3/commits{/sha}","git_commits_url":"https://api.github.com/repos/ml31415/urllib3/git/commits{/sha}","comments_url":"https://api.github.com/repos/ml31415/urllib3/comments{/number}","issue_comment_url":"https://api.github.com/repos/ml31415/urllib3/issues/comments{/number}","contents_url":"https://api.github.com/repos/ml31415/urllib3/contents/{+path}","compare_url":"https://api.github.com/repos/ml31415/urllib3/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ml31415/urllib3/merges","archive_url":"https://api.github.com/repos/ml31415/urllib3/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ml31415/urllib3/downloads","issues_url":"https://api.github.com/repos/ml31415/urllib3/issues{/number}","pulls_url":"https://api.github.com/repos/ml31415/urllib3/pulls{/number}","milestones_url":"https://api.github.com/repos/ml31415/urllib3/milestones{/number}","notifications_url":"https://api.github.com/repos/ml31415/urllib3/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ml31415/urllib3/labels{/name}","releases_url":"https://api.github.com/repos/ml31415/urllib3/releases{/id}","created_at":"2015-02-02T09:48:34Z","updated_at":"2015-02-02T09:48:35Z","pushed_at":"2016-01-11T21:18:11Z","git_url":"git://github.com/ml31415/urllib3.git","ssh_url":"git@github.com:ml31415/urllib3.git","clone_url":"https://github.com/ml31415/urllib3.git","svn_url":"https://github.com/ml31415/urllib3","homepage":"http://urllib3.readthedocs.org/","size":2482,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"shazow:master","ref":"master","sha":"547405c21bad3c72650d71630d31524614cb0812","user":{"login":"shazow","id":6292,"avatar_url":"https://avatars.githubusercontent.com/u/6292?v=3","gravatar_id":"","url":"https://api.github.com/users/shazow","html_url":"https://github.com/shazow","followers_url":"https://api.github.com/users/shazow/followers","following_url":"https://api.github.com/users/shazow/following{/other_user}","gists_url":"https://api.github.com/users/shazow/gists{/gist_id}","starred_url":"https://api.github.com/users/shazow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shazow/subscriptions","organizations_url":"https://api.github.com/users/shazow/orgs","repos_url":"https://api.github.com/users/shazow/repos","events_url":"https://api.github.com/users/shazow/events{/privacy}","received_events_url":"https://api.github.com/users/shazow/received_events","type":"User","site_admin":false},"repo":{"id":2410676,"name":"urllib3","full_name":"shazow/urllib3","owner":{"login":"shazow","id":6292,"avatar_url":"https://avatars.githubusercontent.com/u/6292?v=3","gravatar_id":"","url":"https://api.github.com/users/shazow","html_url":"https://github.com/shazow","followers_url":"https://api.github.com/users/shazow/followers","following_url":"https://api.github.com/users/shazow/following{/other_user}","gists_url":"https://api.github.com/users/shazow/gists{/gist_id}","starred_url":"https://api.github.com/users/shazow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shazow/subscriptions","organizations_url":"https://api.github.com/users/shazow/orgs","repos_url":"https://api.github.com/users/shazow/repos","events_url":"https://api.github.com/users/shazow/events{/privacy}","received_events_url":"https://api.github.com/users/shazow/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/shazow/urllib3","description":"Python HTTP library with thread-safe connection pooling, file post support, sanity friendly, and more.","fork":false,"url":"https://api.github.com/repos/shazow/urllib3","forks_url":"https://api.github.com/repos/shazow/urllib3/forks","keys_url":"https://api.github.com/repos/shazow/urllib3/keys{/key_id}","collaborators_url":"https://api.github.com/repos/shazow/urllib3/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/shazow/urllib3/teams","hooks_url":"https://api.github.com/repos/shazow/urllib3/hooks","issue_events_url":"https://api.github.com/repos/shazow/urllib3/issues/events{/number}","events_url":"https://api.github.com/repos/shazow/urllib3/events","assignees_url":"https://api.github.com/repos/shazow/urllib3/assignees{/user}","branches_url":"https://api.github.com/repos/shazow/urllib3/branches{/branch}","tags_url":"https://api.github.com/repos/shazow/urllib3/tags","blobs_url":"https://api.github.com/repos/shazow/urllib3/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/shazow/urllib3/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/shazow/urllib3/git/refs{/sha}","trees_url":"https://api.github.com/repos/shazow/urllib3/git/trees{/sha}","statuses_url":"https://api.github.com/repos/shazow/urllib3/statuses/{sha}","languages_url":"https://api.github.com/repos/shazow/urllib3/languages","stargazers_url":"https://api.github.com/repos/shazow/urllib3/stargazers","contributors_url":"https://api.github.com/repos/shazow/urllib3/contributors","subscribers_url":"https://api.github.com/repos/shazow/urllib3/subscribers","subscription_url":"https://api.github.com/repos/shazow/urllib3/subscription","commits_url":"https://api.github.com/repos/shazow/urllib3/commits{/sha}","git_commits_url":"https://api.github.com/repos/shazow/urllib3/git/commits{/sha}","comments_url":"https://api.github.com/repos/shazow/urllib3/comments{/number}","issue_comment_url":"https://api.github.com/repos/shazow/urllib3/issues/comments{/number}","contents_url":"https://api.github.com/repos/shazow/urllib3/contents/{+path}","compare_url":"https://api.github.com/repos/shazow/urllib3/compare/{base}...{head}","merges_url":"https://api.github.com/repos/shazow/urllib3/merges","archive_url":"https://api.github.com/repos/shazow/urllib3/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/shazow/urllib3/downloads","issues_url":"https://api.github.com/repos/shazow/urllib3/issues{/number}","pulls_url":"https://api.github.com/repos/shazow/urllib3/pulls{/number}","milestones_url":"https://api.github.com/repos/shazow/urllib3/milestones{/number}","notifications_url":"https://api.github.com/repos/shazow/urllib3/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/shazow/urllib3/labels{/name}","releases_url":"https://api.github.com/repos/shazow/urllib3/releases{/id}","created_at":"2011-09-18T18:08:28Z","updated_at":"2016-01-12T05:52:58Z","pushed_at":"2016-01-11T21:18:11Z","git_url":"git://github.com/shazow/urllib3.git","ssh_url":"git@github.com:shazow/urllib3.git","clone_url":"https://github.com/shazow/urllib3.git","svn_url":"https://github.com/shazow/urllib3","homepage":"http://urllib3.readthedocs.org/","size":4475,"stargazers_count":961,"watchers_count":961,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":299,"mirror_url":null,"open_issues_count":89,"forks":299,"open_issues":89,"watchers":961,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/shazow/urllib3/pulls/775"},"html":{"href":"https://github.com/shazow/urllib3/pull/775"},"issue":{"href":"https://api.github.com/repos/shazow/urllib3/issues/775"},"comments":{"href":"https://api.github.com/repos/shazow/urllib3/issues/775/comments"},"review_comments":{"href":"https://api.github.com/repos/shazow/urllib3/pulls/775/comments"},"review_comment":{"href":"https://api.github.com/repos/shazow/urllib3/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/shazow/urllib3/pulls/775/commits"},"statuses":{"href":"https://api.github.com/repos/shazow/urllib3/statuses/fec75a28b27caaaf6bb546fa9eca9ec8cf2575b6"}}}}