{"action":"closed","issue":{"url":"https://api.github.com/repos/pytorch/text/issues/306","repository_url":"https://api.github.com/repos/pytorch/text","labels_url":"https://api.github.com/repos/pytorch/text/issues/306/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/text/issues/306/comments","events_url":"https://api.github.com/repos/pytorch/text/issues/306/events","html_url":"https://github.com/pytorch/text/issues/306","id":319768834,"node_id":"MDU6SXNzdWUzMTk3Njg4MzQ=","number":306,"title":"MosesTokenizer has been moved out of NLTK due to licensing issues","user":{"login":"alvations","id":1050316,"node_id":"MDQ6VXNlcjEwNTAzMTY=","avatar_url":"https://avatars0.githubusercontent.com/u/1050316?v=4","gravatar_id":"","url":"https://api.github.com/users/alvations","html_url":"https://github.com/alvations","followers_url":"https://api.github.com/users/alvations/followers","following_url":"https://api.github.com/users/alvations/following{/other_user}","gists_url":"https://api.github.com/users/alvations/gists{/gist_id}","starred_url":"https://api.github.com/users/alvations/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alvations/subscriptions","organizations_url":"https://api.github.com/users/alvations/orgs","repos_url":"https://api.github.com/users/alvations/repos","events_url":"https://api.github.com/users/alvations/events{/privacy}","received_events_url":"https://api.github.com/users/alvations/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-03T02:15:42Z","updated_at":"2018-08-07T15:49:51Z","closed_at":"2018-08-07T15:49:51Z","author_association":"NONE","body":"@jekbradbury great work here!\r\n\r\nDue to https://github.com/nltk/nltk/issues/2000, we had to remove MosesTokenizer out of NLTK but now it's hosted on https://github.com/alvations/sacremoses \r\n\r\n```\r\npip install sacremoses\r\n```\r\n\r\nThe silver lining is that the package comes with the data needed for tokenization so there's no need to keep the `nltk_data` directory =)\r\n\r\n----\r\n\r\nI would propose adding `sacremoses` on top of `nltk` because NLTK has another port of a nice tokenizer (by @jonsafari) that people overlook, https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py (I think it's fast too)"}}
{"actor":{"display_login":"mttk"}}