{"action":"created","issue":{"url":"https://api.github.com/repos/shazow/urllib3/issues/717","repository_url":"https://api.github.com/repos/shazow/urllib3","labels_url":"https://api.github.com/repos/shazow/urllib3/issues/717/labels{/name}","comments_url":"https://api.github.com/repos/shazow/urllib3/issues/717/comments","events_url":"https://api.github.com/repos/shazow/urllib3/issues/717/events","html_url":"https://github.com/shazow/urllib3/issues/717","id":111500076,"number":717,"title":"SSL3_WRITE_PENDING error from urllib3.contrib.pyopenssl#sendall","user":{"login":"evnm","id":110020,"avatar_url":"https://avatars.githubusercontent.com/u/110020?v=3","gravatar_id":"","url":"https://api.github.com/users/evnm","html_url":"https://github.com/evnm","followers_url":"https://api.github.com/users/evnm/followers","following_url":"https://api.github.com/users/evnm/following{/other_user}","gists_url":"https://api.github.com/users/evnm/gists{/gist_id}","starred_url":"https://api.github.com/users/evnm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evnm/subscriptions","organizations_url":"https://api.github.com/users/evnm/orgs","repos_url":"https://api.github.com/users/evnm/repos","events_url":"https://api.github.com/users/evnm/events{/privacy}","received_events_url":"https://api.github.com/users/evnm/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-10-14T21:45:59Z","updated_at":"2016-03-31T15:09:33Z","closed_at":"2015-10-17T00:04:41Z","body":"I'm writing a program that makes somewhat large (e.g. tens to hundreds of kilobytes) POST requests over HTTPS using `requests`. Such POSTs fail due to `SSL3_WRITE_PENDING` errors from OpenSSL. In short, `SSL3_WRITE_PENDING` occurs when an incomplete write operation (i.e. a write that resulted in a [`WantWriteError`](http://pyopenssl.sourceforge.net/pyOpenSSL.html/openssl-ssl.html)) is retried with arguments not _exactly_ equal to those passed in the initial call (see [here](http://stackoverflow.com/questions/20715408/solved-pyopenssl-ssl3-write-pendingbad-write-retry-return-self-sslobj-write) and [here](http://stackoverflow.com/questions/2997218/why-am-i-getting-error1409f07fssl-routinesssl3-write-pending-bad-write-retr)).\r\n\r\nStracktrace from my call into `requests` down through `urllib3`:\r\n```\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/sessions.py\", line 511, in post\r\nreturn self.request('POST', url, data=data, json=json, **kwargs)\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/sessions.py\", line 468, in request\r\nresp = self.send(prep, **send_kwargs)\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/sessions.py\", line 576, in send\r\nr = adapter.send(request, **kwargs)\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/adapters.py\", line 370, in send\r\ntimeout=timeout\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py\", line 559, in urlopen\r\nbody=body, headers=headers)\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py\", line 353, in _make_request\r\nconn.request(method, url, **httplib_request_kw)\r\nFile \"/usr/local/lib/python2.7/httplib.py\", line 1053, in request\r\nself._send_request(method, url, body, headers)\r\nFile \"/usr/local/lib/python2.7/httplib.py\", line 1093, in _send_request\r\nself.endheaders(body)\r\nFile \"/usr/local/lib/python2.7/httplib.py\", line 1049, in endheaders\r\nself._send_output(message_body)\r\nFile \"/usr/local/lib/python2.7/httplib.py\", line 893, in _send_output\r\nself.send(msg)\r\nFile \"/usr/local/lib/python2.7/httplib.py\", line 869, in send\r\nself.sock.sendall(data)\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py\", line 220, in sendall\r\nsent = self._send_until_done(data[total_sent:total_sent+SSL_WRITE_BLOCKSIZE])\r\nFile \"/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py\", line 206, in _send_until_done\r\nreturn self.connection.send(data)\r\nFile \"/usr/local/lib/python2.7/site-packages/OpenSSL/SSL.py\", line 1271, in send\r\nself._raise_ssl_error(self._ssl, result)\r\nFile \"/usr/local/lib/python2.7/site-packages/OpenSSL/SSL.py\", line 1187, in _raise_ssl_error\r\n_raise_current_error()\r\nFile \"/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py\", line 48, in exception_from_error_queue\r\nraise exception_type(errors)\r\nError: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]\r\n```\r\n\r\nA bit of digging revealed #412 and #626 as issues that relate to the relevant code in urllib3. `WrappedSocket.sendall` proxies to `WrappedSocket._send_until_done`, which invokes [`OpenSSL.SSL.send`](https://github.com/pyca/pyopenssl/blob/8adee5e7f8759886678ab83919d5956151e825f3/OpenSSL/SSL.py#L1249) in a loop. The conditions for loop termination are a) the write succeeds or b) a timeout expires while waiting for the socket to become writable in the case of a `WantWriteError`.\r\n\r\nI think that this issue boils down to:\r\n\r\n1. `WrappedSocket.sendall` converts the data to be sent to a `memoryview`.\r\n2. `WrappedSocket._send_until_done` invokes `OpenSSL.SSL.send` in a loop according to the conditions above.\r\n3. [`OpenSSL.SSL.send` calls the `memoryview.tobytes()`](https://github.com/pyca/pyopenssl/blob/8adee5e7f8759886678ab83919d5956151e825f3/OpenSSL/SSL.py#L1264) and passes the bytestring result in the OpenSSL write call.\r\n4. In the `WantWriteError` case, 2 and 3 are repeated and a _new_ bytestring is passed to the OpenSSL write, resulting in a `SSL3_WRITE_PENDING` error.\r\n\r\nI was able to get around this issue by patching `urllib3/contrib/pyopenssl.py` to enforce that a single bytestring is used for all calls to `OpenSSL.SSL.send`: https://gist.github.com/evnm/af92092c6a7776e08339\r\n\r\nPlease advise on whether this is a good way to fix the issue. I tried adding a test case to `test/with_dummyserver/test_https.py`, but haven't figured out how to tickle the specific issue I've run into.\r\n\r\nRelevant versions in use:\r\n* Python 2.7.10\r\n* requests 2.8.1\r\n* pyOpenSSL 0.15.1\r\n* urllib3 1.12"},"comment":{"url":"https://api.github.com/repos/shazow/urllib3/issues/comments/203979834","html_url":"https://github.com/shazow/urllib3/issues/717#issuecomment-203979834","issue_url":"https://api.github.com/repos/shazow/urllib3/issues/717","id":203979834,"user":{"login":"Lukasa","id":1382556,"avatar_url":"https://avatars.githubusercontent.com/u/1382556?v=3","gravatar_id":"","url":"https://api.github.com/users/Lukasa","html_url":"https://github.com/Lukasa","followers_url":"https://api.github.com/users/Lukasa/followers","following_url":"https://api.github.com/users/Lukasa/following{/other_user}","gists_url":"https://api.github.com/users/Lukasa/gists{/gist_id}","starred_url":"https://api.github.com/users/Lukasa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lukasa/subscriptions","organizations_url":"https://api.github.com/users/Lukasa/orgs","repos_url":"https://api.github.com/users/Lukasa/repos","events_url":"https://api.github.com/users/Lukasa/events{/privacy}","received_events_url":"https://api.github.com/users/Lukasa/received_events","type":"User","site_admin":false},"created_at":"2016-03-31T15:09:33Z","updated_at":"2016-03-31T15:09:33Z","body":"@nitzmahone Does this work reproducibly in the same virtual environment just repeatedly POSTing large payloads (e.g. 800kB of data) to httpbin.org?"}}