{"action":"created","issue":{"url":"https://api.github.com/repos/iclab/centinel/issues/257","repository_url":"https://api.github.com/repos/iclab/centinel","labels_url":"https://api.github.com/repos/iclab/centinel/issues/257/labels{/name}","comments_url":"https://api.github.com/repos/iclab/centinel/issues/257/comments","events_url":"https://api.github.com/repos/iclab/centinel/issues/257/events","html_url":"https://github.com/iclab/centinel/issues/257","id":165674907,"number":257,"title":"encoding exception when storing results","user":{"login":"rpanah","id":6577290,"avatar_url":"https://avatars.githubusercontent.com/u/6577290?v=3","gravatar_id":"","url":"https://api.github.com/users/rpanah","html_url":"https://github.com/rpanah","followers_url":"https://api.github.com/users/rpanah/followers","following_url":"https://api.github.com/users/rpanah/following{/other_user}","gists_url":"https://api.github.com/users/rpanah/gists{/gist_id}","starred_url":"https://api.github.com/users/rpanah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rpanah/subscriptions","organizations_url":"https://api.github.com/users/rpanah/orgs","repos_url":"https://api.github.com/users/rpanah/repos","events_url":"https://api.github.com/users/rpanah/events{/privacy}","received_events_url":"https://api.github.com/users/rpanah/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/iclab/centinel/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-14T22:41:01Z","updated_at":"2016-09-04T15:04:06Z","closed_at":null,"body":"```\r\n2016-07-14 18:34:29,043 client.py(line 428) ERROR: Error saving results for baseline to file: 'utf8' codec can't decode byte 0xe0 in position 3: invalid continuation byte\r\nTraceback (most recent call last):\r\n  File \"/Users/abbas/iclab/centinel/centinel/client.py\", line 420, in run_exp\r\n    json.dump(results, result_file, indent=2, separators=(',', ': '))\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py\", line 189, in dump\r\n    for chunk in iterable:\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 434, in _iterencode\r\n    for chunk in _iterencode_dict(o, _current_indent_level):\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 408, in _iterencode_dict\r\n    for chunk in chunks:\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 332, in _iterencode_list\r\n    for chunk in chunks:\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 408, in _iterencode_dict\r\n    for chunk in chunks:\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 408, in _iterencode_dict\r\n    for chunk in chunks:\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 408, in _iterencode_dict\r\n    for chunk in chunks:\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 408, in _iterencode_dict\r\n    for chunk in chunks:\r\n  File \"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py\", line 390, in _iterencode_dict\r\n    yield _encoder(value)\r\nUnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 3: invalid continuation byte\r\n```"},"comment":{"url":"https://api.github.com/repos/iclab/centinel/issues/comments/244608438","html_url":"https://github.com/iclab/centinel/issues/257#issuecomment-244608438","issue_url":"https://api.github.com/repos/iclab/centinel/issues/257","id":244608438,"user":{"login":"rpanah","id":6577290,"avatar_url":"https://avatars.githubusercontent.com/u/6577290?v=3","gravatar_id":"","url":"https://api.github.com/users/rpanah","html_url":"https://github.com/rpanah","followers_url":"https://api.github.com/users/rpanah/followers","following_url":"https://api.github.com/users/rpanah/following{/other_user}","gists_url":"https://api.github.com/users/rpanah/gists{/gist_id}","starred_url":"https://api.github.com/users/rpanah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rpanah/subscriptions","organizations_url":"https://api.github.com/users/rpanah/orgs","repos_url":"https://api.github.com/users/rpanah/repos","events_url":"https://api.github.com/users/rpanah/events{/privacy}","received_events_url":"https://api.github.com/users/rpanah/received_events","type":"User","site_admin":false},"created_at":"2016-09-04T15:04:05Z","updated_at":"2016-09-04T15:04:05Z","body":"This problem was particularly hard to reproduce since it is fairly uncommon (1 in ~2000 websites) to have non-UTF-8 characters in the headers (they're usually simple ASCII), and even harder to diagnose since the encoder will not say exactly where in the huge JSON object the error occurs (the example below was taken out of a 90MB file with ~0.5 million lines of data).\r\n\r\nThe encoding issue here stems from non-unicode characters in HTTP request/response headers. The UTF-8 decoder can't decode the non-Unicode characters and will cause the entire result file to fail to write to disk.\r\n\r\nThe solution that seems to work, would be to ignore encoding errors (`ensure_ascii=False`) when writing the JSON object to file:\r\n`                json.dump(results, result_file, indent=2, separators=(',', ': '), ensure_ascii=False)`\r\n\r\nDoing this will break server-side analysis for results that have non-Unicode characters (JSON decoding will fail), so we will need to handle (e.g. replace) non-decodable characters on the server when analyzing results.\r\n\r\nHere's an example with a non-UTF-8 character in a cookie:\r\n\r\n```\r\n        \"http://www.zoomshare.com\": {\r\n          \"request\": {\r\n            \"headers\": {\r\n              \"User-Agent\": \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36\"\r\n            },\r\n            \"path\": \"/\",\r\n            \"host\": \"www.zoomshare.com\",\r\n            \"method\": \"GET\",\r\n            \"ssl\": false\r\n          },\r\n          \"response\": {\r\n            \"status\": 200,\r\n            \"headers\": {\r\n              \"Transfer-Encoding\": \"chunked\",\r\n              \"Set-Cookie\": \"dw=†IÃ; path=/\",\r\n              \"Server\": \"Apache/1.3.41 (Unix) mod_perl/1.30 mod_fastcgi/2.4.6\",\r\n              \"Last-Modified\": \"Thu, 04 Sep 2014 19:54:14 GMT\",\r\n              \"Connection\": \"close\",\r\n              \"Date\": \"Sat, 03 Sep 2016 20:45:10 GMT\",\r\n              \"Content-Type\": \"text/html; charset=iso-8859-1\"\r\n            },\r\n            \"reason\": \"OK\",\r\n            \"body\": \"[...]\"\r\n          }\r\n        }\r\n```\r\n"}}
{"actor":{"display_login":"rpanah"}}