{"action":"created","issue":{"url":"https://api.github.com/repos/benoitc/gunicorn/issues/1778","repository_url":"https://api.github.com/repos/benoitc/gunicorn","labels_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1778/labels{/name}","comments_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1778/comments","events_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1778/events","html_url":"https://github.com/benoitc/gunicorn/issues/1778","id":321352180,"node_id":"MDU6SXNzdWUzMjEzNTIxODA=","number":1778,"title":"Encoding headers with ASCII breaks applications expecting them to be encoded in latin-1","user":{"login":"jrast","id":146369,"node_id":"MDQ6VXNlcjE0NjM2OQ==","avatar_url":"https://avatars3.githubusercontent.com/u/146369?v=4","gravatar_id":"","url":"https://api.github.com/users/jrast","html_url":"https://github.com/jrast","followers_url":"https://api.github.com/users/jrast/followers","following_url":"https://api.github.com/users/jrast/following{/other_user}","gists_url":"https://api.github.com/users/jrast/gists{/gist_id}","starred_url":"https://api.github.com/users/jrast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrast/subscriptions","organizations_url":"https://api.github.com/users/jrast/orgs","repos_url":"https://api.github.com/users/jrast/repos","events_url":"https://api.github.com/users/jrast/events{/privacy}","received_events_url":"https://api.github.com/users/jrast/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":26,"created_at":"2018-05-08T21:24:07Z","updated_at":"2018-11-09T00:51:22Z","closed_at":null,"author_association":"NONE","body":"Applications, like flask, which expect that the headers are using latin-1 encoding as specified in [rfc5987] will break gunicorn because gunicorn uses ASCII encoding.\r\n\r\nIssue #1353 was closed with the comment\r\n> I'm going to close this as it's well documented around the web that HTTP headers should be ASCII, unfortunately.\r\n\r\nwhich is, according to [rfc5987] wrong. Headers should be encoded ISO-8859-1 (aka latin-1). However, a comment in the issue also mentioned commit 5f4ebd2eb2b08783a5fbefe79d09fcb3fc1fbc73, in which the encoding of the headers was switched back from latin-1 to ascii based on a section in [rfc7230] which states that US-ASCII should be used:\r\n\r\n> Historically, HTTP has allowed field content with text in the\r\n   ISO-8859-1 charset [ISO-8859-1], supporting other charsets only\r\n   through use of [RFC2047] encoding.  In practice, most HTTP header\r\n   field values use only a subset of the US-ASCII charset [USASCII].\r\n   Newly defined header fields SHOULD limit their field values to\r\n   US-ASCII octets.  [...]\r\n\r\nNow, I'm not into all this RFC specifications and don't know which is the right way to do it. However, as other applications expect latin-1 as specified in [rfc5987] but gunicorn implements the version given in [rfc7230], it might be worth to discuss this, even if it's just for a clarification on which track gunicorn is.\r\n\r\nRelated Issues in gunicorn:\r\nhttps://github.com/benoitc/gunicorn/issues/1353 : wsgi.py > send_headers: encoding problem.\r\n\r\nRelated Issues in Flask:\r\nhttps://github.com/pallets/flask/issues/2766 : send_file: latin-1 encoding not compatible with gunicorn\r\nhttps://github.com/pallets/flask/pull/2223 : Fix send_file's attachment_filename to work with non-ascii filenames\r\n\r\nRelated RFC:\r\nhttps://tools.ietf.org/html/rfc5987#section-1\r\n\r\n[rfc5987]:https://tools.ietf.org/html/rfc5987#section-1\r\n[rfc7230]:https://tools.ietf.org/html/rfc7230#section-3.2.4"},"comment":{"url":"https://api.github.com/repos/benoitc/gunicorn/issues/comments/437210603","html_url":"https://github.com/benoitc/gunicorn/issues/1778#issuecomment-437210603","issue_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1778","id":437210603,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzIxMDYwMw==","user":{"login":"javabrett","id":1103477,"node_id":"MDQ6VXNlcjExMDM0Nzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1103477?v=4","gravatar_id":"","url":"https://api.github.com/users/javabrett","html_url":"https://github.com/javabrett","followers_url":"https://api.github.com/users/javabrett/followers","following_url":"https://api.github.com/users/javabrett/following{/other_user}","gists_url":"https://api.github.com/users/javabrett/gists{/gist_id}","starred_url":"https://api.github.com/users/javabrett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javabrett/subscriptions","organizations_url":"https://api.github.com/users/javabrett/orgs","repos_url":"https://api.github.com/users/javabrett/repos","events_url":"https://api.github.com/users/javabrett/events{/privacy}","received_events_url":"https://api.github.com/users/javabrett/received_events","type":"User","site_admin":false},"created_at":"2018-11-09T00:51:22Z","updated_at":"2018-11-09T00:51:22Z","author_association":"CONTRIBUTOR","body":"Some notes on the behaviour differences between Werkzeug(+Flask) and Gunicorn(+Flask):\r\n\r\nApplication attempts to set header value|Werkzeug/0.14.1+Flask|Gunicorn 19.9/master\r\n---------------------------------------- | ---------------------- | --------------------\r\n`b'f\\xc3\\xb6o'.decode(\"utf-8\", \"strict\") = \"föo\"`|**200** OK, header value is sent, contains bytes `\\x66\\xf6\\x6f = föo`|**500**, `UnicodeEncodeError: 'ascii' codec can't encode character '\\xf6' in position 164: ordinal not in range(128)`\r\n`b'Swan: \\xF0\\x9F\\xA6\\xA2'.decode(\"utf-8\", \"strict\") = \"Swan: (swan glyph)\"`|**200** but **empty response**, errors out while encoding headers - `UnicodeEncodeError: 'latin-1' codec can't encode character '\\U0001f9a2' in position 19: ordinal not in range(256)`|**500**, `UnicodeEncodeError: 'ascii' codec can't encode character '\\U0001f9a2' in position 169: ordinal not in range(128)`\r\n\r\nNote that when writing-out the response headers, Werkzeug delegates to Python builtin `BaseHTTPRequestHandler`:\r\n\r\nhttps://github.com/pallets/werkzeug/blob/7477be2853df70a022d9613e765581b9411c3c39/werkzeug/serving.py#L235\r\n\r\n... which uses `latin-1`:\r\n\r\nhttps://github.com/python/cpython/blob/fd512d76456b65c529a5bc58d8cfe73e4a10de7a/Lib/http/server.py#L516\r\n\r\nThat will explain the behaviour differences.\r\n\r\nMy reading of the considerable number and history of RFCs on this suggests that applications and servers should be restricting themselves to `US-ASCII`, but that (perhaps for historical reasons) latin-1 is allowed, but clients should treat such octets as opaque.\r\n\r\nEither way, ideally Gunicorn and Werkzeug should behave the same here if possible, and it seems that a latin-1 character should not error the response.  Gunicorn has a superior way of serializing the header values, so if there is a non-latin-1 character present, it detects this before committing 200 OK (like happens in Werkzeug), and can send 500.  But it appears to me that it should allow `latin-1`."}}
{"actor":{"display_login":"javabrett"}}