{"action":"created","comment":{"url":"https://api.github.com/repos/StevenBlack/hosts/pulls/comments/171971716","pull_request_review_id":100924443,"id":171971716,"diff_hunk":"@@ -1126,6 +1129,60 @@ def remove_old_hosts_file(backup):\n # End File Logic\n \n \n+def domain_to_idna(line):\n+    \"\"\"\n+    Encode a domain which is presente into a line into `idna`. This way we\n+    avoid the most encoding issue.\n+\n+    Parameters\n+    ----------\n+    line : str\n+        The line we have to encode/decode.\n+\n+    Returns\n+    -------\n+    line : str\n+        The line in a converted format.\n+\n+    Notes\n+    -----\n+    - This method/function encode only the domain to `idna` format because in\n+        most cases the encoding issue is due to a domain which looks like\n+        `b'\\xc9\\xa2oogle.com'.decode('idna')`.\n+    - About the splitting:\n+        We split because we only want to encode the domain and not the full\n+            line which may cause some issue. Keep in mind that we split but we\n+            still concatenate once we encoded the domain.","path":"updateHostsFile.py","position":62,"original_position":62,"commit_id":"d98b31fb921ff35fb1a421243fa4876504cf676e","original_commit_id":"d98b31fb921ff35fb1a421243fa4876504cf676e","user":{"login":"gfyoung","id":9273653,"avatar_url":"https://avatars0.githubusercontent.com/u/9273653?v=4","gravatar_id":"","url":"https://api.github.com/users/gfyoung","html_url":"https://github.com/gfyoung","followers_url":"https://api.github.com/users/gfyoung/followers","following_url":"https://api.github.com/users/gfyoung/following{/other_user}","gists_url":"https://api.github.com/users/gfyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/gfyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gfyoung/subscriptions","organizations_url":"https://api.github.com/users/gfyoung/orgs","repos_url":"https://api.github.com/users/gfyoung/repos","events_url":"https://api.github.com/users/gfyoung/events{/privacy}","received_events_url":"https://api.github.com/users/gfyoung/received_events","type":"User","site_admin":false},"body":"Grammar nits:\r\n\r\n\"line, which may cause some issues\" (add comma + pluralize issue)\r\n\"split, but we still\" (add comma after 'split')","created_at":"2018-03-02T21:48:35Z","updated_at":"2018-03-02T21:48:35Z","html_url":"https://github.com/StevenBlack/hosts/pull/520#discussion_r171971716","pull_request_url":"https://api.github.com/repos/StevenBlack/hosts/pulls/520","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/StevenBlack/hosts/pulls/comments/171971716"},"html":{"href":"https://github.com/StevenBlack/hosts/pull/520#discussion_r171971716"},"pull_request":{"href":"https://api.github.com/repos/StevenBlack/hosts/pulls/520"}},"in_reply_to_id":171971481},"pull_request":{"url":"https://api.github.com/repos/StevenBlack/hosts/pulls/520","id":172104108,"html_url":"https://github.com/StevenBlack/hosts/pull/520","diff_url":"https://github.com/StevenBlack/hosts/pull/520.diff","patch_url":"https://github.com/StevenBlack/hosts/pull/520.patch","issue_url":"https://api.github.com/repos/StevenBlack/hosts/issues/520","number":520,"state":"open","locked":false,"title":"Possible fix of the encoding and/or downlaod issue(s)","user":{"login":"funilrys","id":14127285,"avatar_url":"https://avatars0.githubusercontent.com/u/14127285?v=4","gravatar_id":"","url":"https://api.github.com/users/funilrys","html_url":"https://github.com/funilrys","followers_url":"https://api.github.com/users/funilrys/followers","following_url":"https://api.github.com/users/funilrys/following{/other_user}","gists_url":"https://api.github.com/users/funilrys/gists{/gist_id}","starred_url":"https://api.github.com/users/funilrys/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/funilrys/subscriptions","organizations_url":"https://api.github.com/users/funilrys/orgs","repos_url":"https://api.github.com/users/funilrys/repos","events_url":"https://api.github.com/users/funilrys/events{/privacy}","received_events_url":"https://api.github.com/users/funilrys/received_events","type":"User","site_admin":false},"body":"Hello, to all member of this community!\r\n\r\nAfter a long time of commenting, watching, reading and learning from issues posted here it's time for me to propose the following patches/commits.\r\n\r\nIt's been a long time since I wanted to search a possible solution to this recurrent issue and here's my proposition.\r\n\r\nBasically, since I consider that many encoding issues are because upstream maintainer does not convert/encode the domains which rely on [IDN homograph attack](https://en.wikipedia.org/wiki/IDN_homograph_attack), I decided to read all line and convert/encode all domain using `str.encode('IDNA')` (I do not change original structure). \r\n\r\nBecause my main objective was to fix @notDavid's [protocol](https://github.com/StevenBlack/hosts/issues/514#issuecomment-368932152) which is the only one I was able to reproduce, I introduced `BeautifulSoup()` to do the decoding of the downloaded data before we read each line and encode/decode.\r\n\r\n## About the tests\r\n\r\nBefore anything, sorry for adding the `fix tests issues` commits but I wanted to fix those before submitting my PR. You can find the tests I used as the reference to committing those change at https://travis-ci.org/funilrys/hosts.\r\n\r\nPlease note that I have to submit an issue to upstream but if you see the following or related when runing the tests under `Python >= 3` it's not a local issue. \r\n```\r\n/home/travis/miniconda3/envs/hosts/lib/python3.5/site-packages/bs4/builder/_lxml.py:250: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead\r\n```\r\n\r\n## About `requirements*.txt`\r\n\r\nThis submission also introduces `requirements.txt` files which I generally use to install dependencies with `pip install -r requirements.txt` (I'm almost everytime under `virtualenv`).\r\n\r\nAs I did not get an answer ([cf](https://github.com/StevenBlack/hosts/issues/514#issuecomment-368983762)) I took the initiative to includes them anyway.\r\n\r\nPlease note that the `requirements*.txt` files were generated by [pipreqs](https://github.com/bndr/pipreqs).\r\n\r\n## So to quote my commit message\r\n\r\nPlease note that this patch also introduces `domain_to_idna()`\r\nwhich is in charge of converting a domain in a line into\r\nIDNA and/or UTF-8 format.\r\n\r\nAlso, note the introduction of BeautifulSoup() which helps\r\nus to decode data from the downloaded URL.\r\n\r\nFixes (issue(s)/protocol(s) I was able to reproduce):\r\n  * https://github.com/StevenBlack/hosts/issues/514#issuecomment-368932152\r\n    \r\nPossible fix of (issue(s)/protocol(s) I wasn't able to reproduce):\r\n  * https://github.com/StevenBlack/hosts/issues/514#issue-300048106\r\n  * https://github.com/StevenBlack/hosts/issues/494#issue-296166492\r\n  * https://github.com/StevenBlack/hosts/issues/420#issue-267453114\r\n  * https://github.com/StevenBlack/hosts/issues/372#issue-246927047\r\n  * https://github.com/StevenBlack/hosts/issues/382#issuecomment-322010562\r\n\r\n## Footnote(s)\r\n\r\nSorry for the typo or bad English structure in my commits, code or this description but English is still a language I learn.\r\n\r\nSo I'm open to any pieces of advice and others around my English, coding style or other :smile_cat: \r\n\r\nCheers,\r\nNissar.","created_at":"2018-02-28T23:30:40Z","updated_at":"2018-03-02T21:48:35Z","closed_at":null,"merged_at":null,"merge_commit_sha":"450b5daab3e992ee31077f615e2d9c90da2958e0","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/StevenBlack/hosts/pulls/520/commits","review_comments_url":"https://api.github.com/repos/StevenBlack/hosts/pulls/520/comments","review_comment_url":"https://api.github.com/repos/StevenBlack/hosts/pulls/comments{/number}","comments_url":"https://api.github.com/repos/StevenBlack/hosts/issues/520/comments","statuses_url":"https://api.github.com/repos/StevenBlack/hosts/statuses/d98b31fb921ff35fb1a421243fa4876504cf676e","head":{"label":"funilrys:notDavid-protocol","ref":"notDavid-protocol","sha":"d98b31fb921ff35fb1a421243fa4876504cf676e","user":{"login":"funilrys","id":14127285,"avatar_url":"https://avatars0.githubusercontent.com/u/14127285?v=4","gravatar_id":"","url":"https://api.github.com/users/funilrys","html_url":"https://github.com/funilrys","followers_url":"https://api.github.com/users/funilrys/followers","following_url":"https://api.github.com/users/funilrys/following{/other_user}","gists_url":"https://api.github.com/users/funilrys/gists{/gist_id}","starred_url":"https://api.github.com/users/funilrys/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/funilrys/subscriptions","organizations_url":"https://api.github.com/users/funilrys/orgs","repos_url":"https://api.github.com/users/funilrys/repos","events_url":"https://api.github.com/users/funilrys/events{/privacy}","received_events_url":"https://api.github.com/users/funilrys/received_events","type":"User","site_admin":false},"repo":{"id":123032171,"name":"hosts","full_name":"funilrys/hosts","owner":{"login":"funilrys","id":14127285,"avatar_url":"https://avatars0.githubusercontent.com/u/14127285?v=4","gravatar_id":"","url":"https://api.github.com/users/funilrys","html_url":"https://github.com/funilrys","followers_url":"https://api.github.com/users/funilrys/followers","following_url":"https://api.github.com/users/funilrys/following{/other_user}","gists_url":"https://api.github.com/users/funilrys/gists{/gist_id}","starred_url":"https://api.github.com/users/funilrys/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/funilrys/subscriptions","organizations_url":"https://api.github.com/users/funilrys/orgs","repos_url":"https://api.github.com/users/funilrys/repos","events_url":"https://api.github.com/users/funilrys/events{/privacy}","received_events_url":"https://api.github.com/users/funilrys/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/funilrys/hosts","description":"Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others.  You can optionally invoke extensions to block additional sites by category. ","fork":true,"url":"https://api.github.com/repos/funilrys/hosts","forks_url":"https://api.github.com/repos/funilrys/hosts/forks","keys_url":"https://api.github.com/repos/funilrys/hosts/keys{/key_id}","collaborators_url":"https://api.github.com/repos/funilrys/hosts/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/funilrys/hosts/teams","hooks_url":"https://api.github.com/repos/funilrys/hosts/hooks","issue_events_url":"https://api.github.com/repos/funilrys/hosts/issues/events{/number}","events_url":"https://api.github.com/repos/funilrys/hosts/events","assignees_url":"https://api.github.com/repos/funilrys/hosts/assignees{/user}","branches_url":"https://api.github.com/repos/funilrys/hosts/branches{/branch}","tags_url":"https://api.github.com/repos/funilrys/hosts/tags","blobs_url":"https://api.github.com/repos/funilrys/hosts/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/funilrys/hosts/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/funilrys/hosts/git/refs{/sha}","trees_url":"https://api.github.com/repos/funilrys/hosts/git/trees{/sha}","statuses_url":"https://api.github.com/repos/funilrys/hosts/statuses/{sha}","languages_url":"https://api.github.com/repos/funilrys/hosts/languages","stargazers_url":"https://api.github.com/repos/funilrys/hosts/stargazers","contributors_url":"https://api.github.com/repos/funilrys/hosts/contributors","subscribers_url":"https://api.github.com/repos/funilrys/hosts/subscribers","subscription_url":"https://api.github.com/repos/funilrys/hosts/subscription","commits_url":"https://api.github.com/repos/funilrys/hosts/commits{/sha}","git_commits_url":"https://api.github.com/repos/funilrys/hosts/git/commits{/sha}","comments_url":"https://api.github.com/repos/funilrys/hosts/comments{/number}","issue_comment_url":"https://api.github.com/repos/funilrys/hosts/issues/comments{/number}","contents_url":"https://api.github.com/repos/funilrys/hosts/contents/{+path}","compare_url":"https://api.github.com/repos/funilrys/hosts/compare/{base}...{head}","merges_url":"https://api.github.com/repos/funilrys/hosts/merges","archive_url":"https://api.github.com/repos/funilrys/hosts/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/funilrys/hosts/downloads","issues_url":"https://api.github.com/repos/funilrys/hosts/issues{/number}","pulls_url":"https://api.github.com/repos/funilrys/hosts/pulls{/number}","milestones_url":"https://api.github.com/repos/funilrys/hosts/milestones{/number}","notifications_url":"https://api.github.com/repos/funilrys/hosts/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/funilrys/hosts/labels{/name}","releases_url":"https://api.github.com/repos/funilrys/hosts/releases{/id}","deployments_url":"https://api.github.com/repos/funilrys/hosts/deployments","created_at":"2018-02-26T21:21:49Z","updated_at":"2018-02-26T21:21:51Z","pushed_at":"2018-03-02T21:43:31Z","git_url":"git://github.com/funilrys/hosts.git","ssh_url":"git@github.com:funilrys/hosts.git","clone_url":"https://github.com/funilrys/hosts.git","svn_url":"https://github.com/funilrys/hosts","homepage":"","size":61854,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"StevenBlack:master","ref":"master","sha":"37da299efb31a589e37db092abb805d759b477c3","user":{"login":"StevenBlack","id":80144,"avatar_url":"https://avatars0.githubusercontent.com/u/80144?v=4","gravatar_id":"","url":"https://api.github.com/users/StevenBlack","html_url":"https://github.com/StevenBlack","followers_url":"https://api.github.com/users/StevenBlack/followers","following_url":"https://api.github.com/users/StevenBlack/following{/other_user}","gists_url":"https://api.github.com/users/StevenBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/StevenBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StevenBlack/subscriptions","organizations_url":"https://api.github.com/users/StevenBlack/orgs","repos_url":"https://api.github.com/users/StevenBlack/repos","events_url":"https://api.github.com/users/StevenBlack/events{/privacy}","received_events_url":"https://api.github.com/users/StevenBlack/received_events","type":"User","site_admin":false},"repo":{"id":4008931,"name":"hosts","full_name":"StevenBlack/hosts","owner":{"login":"StevenBlack","id":80144,"avatar_url":"https://avatars0.githubusercontent.com/u/80144?v=4","gravatar_id":"","url":"https://api.github.com/users/StevenBlack","html_url":"https://github.com/StevenBlack","followers_url":"https://api.github.com/users/StevenBlack/followers","following_url":"https://api.github.com/users/StevenBlack/following{/other_user}","gists_url":"https://api.github.com/users/StevenBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/StevenBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StevenBlack/subscriptions","organizations_url":"https://api.github.com/users/StevenBlack/orgs","repos_url":"https://api.github.com/users/StevenBlack/repos","events_url":"https://api.github.com/users/StevenBlack/events{/privacy}","received_events_url":"https://api.github.com/users/StevenBlack/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/StevenBlack/hosts","description":"Extending and consolidating hosts files from several well-curated sources like adaway.org, mvps.org, malwaredomainlist.com, someonewhocares.org, and potentially others.  You can optionally invoke extensions to block additional sites by category. ","fork":false,"url":"https://api.github.com/repos/StevenBlack/hosts","forks_url":"https://api.github.com/repos/StevenBlack/hosts/forks","keys_url":"https://api.github.com/repos/StevenBlack/hosts/keys{/key_id}","collaborators_url":"https://api.github.com/repos/StevenBlack/hosts/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/StevenBlack/hosts/teams","hooks_url":"https://api.github.com/repos/StevenBlack/hosts/hooks","issue_events_url":"https://api.github.com/repos/StevenBlack/hosts/issues/events{/number}","events_url":"https://api.github.com/repos/StevenBlack/hosts/events","assignees_url":"https://api.github.com/repos/StevenBlack/hosts/assignees{/user}","branches_url":"https://api.github.com/repos/StevenBlack/hosts/branches{/branch}","tags_url":"https://api.github.com/repos/StevenBlack/hosts/tags","blobs_url":"https://api.github.com/repos/StevenBlack/hosts/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/StevenBlack/hosts/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/StevenBlack/hosts/git/refs{/sha}","trees_url":"https://api.github.com/repos/StevenBlack/hosts/git/trees{/sha}","statuses_url":"https://api.github.com/repos/StevenBlack/hosts/statuses/{sha}","languages_url":"https://api.github.com/repos/StevenBlack/hosts/languages","stargazers_url":"https://api.github.com/repos/StevenBlack/hosts/stargazers","contributors_url":"https://api.github.com/repos/StevenBlack/hosts/contributors","subscribers_url":"https://api.github.com/repos/StevenBlack/hosts/subscribers","subscription_url":"https://api.github.com/repos/StevenBlack/hosts/subscription","commits_url":"https://api.github.com/repos/StevenBlack/hosts/commits{/sha}","git_commits_url":"https://api.github.com/repos/StevenBlack/hosts/git/commits{/sha}","comments_url":"https://api.github.com/repos/StevenBlack/hosts/comments{/number}","issue_comment_url":"https://api.github.com/repos/StevenBlack/hosts/issues/comments{/number}","contents_url":"https://api.github.com/repos/StevenBlack/hosts/contents/{+path}","compare_url":"https://api.github.com/repos/StevenBlack/hosts/compare/{base}...{head}","merges_url":"https://api.github.com/repos/StevenBlack/hosts/merges","archive_url":"https://api.github.com/repos/StevenBlack/hosts/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/StevenBlack/hosts/downloads","issues_url":"https://api.github.com/repos/StevenBlack/hosts/issues{/number}","pulls_url":"https://api.github.com/repos/StevenBlack/hosts/pulls{/number}","milestones_url":"https://api.github.com/repos/StevenBlack/hosts/milestones{/number}","notifications_url":"https://api.github.com/repos/StevenBlack/hosts/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/StevenBlack/hosts/labels{/name}","releases_url":"https://api.github.com/repos/StevenBlack/hosts/releases{/id}","deployments_url":"https://api.github.com/repos/StevenBlack/hosts/deployments","created_at":"2012-04-12T20:22:50Z","updated_at":"2018-03-02T21:37:05Z","pushed_at":"2018-03-02T21:43:31Z","git_url":"git://github.com/StevenBlack/hosts.git","ssh_url":"git@github.com:StevenBlack/hosts.git","clone_url":"https://github.com/StevenBlack/hosts.git","svn_url":"https://github.com/StevenBlack/hosts","homepage":"","size":61800,"stargazers_count":8274,"watchers_count":8274,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":772,"mirror_url":null,"archived":false,"open_issues_count":58,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":772,"open_issues":58,"watchers":8274,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/StevenBlack/hosts/pulls/520"},"html":{"href":"https://github.com/StevenBlack/hosts/pull/520"},"issue":{"href":"https://api.github.com/repos/StevenBlack/hosts/issues/520"},"comments":{"href":"https://api.github.com/repos/StevenBlack/hosts/issues/520/comments"},"review_comments":{"href":"https://api.github.com/repos/StevenBlack/hosts/pulls/520/comments"},"review_comment":{"href":"https://api.github.com/repos/StevenBlack/hosts/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/StevenBlack/hosts/pulls/520/commits"},"statuses":{"href":"https://api.github.com/repos/StevenBlack/hosts/statuses/d98b31fb921ff35fb1a421243fa4876504cf676e"}},"author_association":"NONE"}}
{"actor":{"display_login":"gfyoung"}}