{"action":"opened","issue":{"url":"https://api.github.com/repos/explosion/spaCy/issues/685","repository_url":"https://api.github.com/repos/explosion/spaCy","labels_url":"https://api.github.com/repos/explosion/spaCy/issues/685/labels{/name}","comments_url":"https://api.github.com/repos/explosion/spaCy/issues/685/comments","events_url":"https://api.github.com/repos/explosion/spaCy/issues/685/events","html_url":"https://github.com/explosion/spaCy/issues/685","id":195550702,"number":685,"title":"Unicode apostrophe confuses tokenizer","user":{"login":"pokey","id":755842,"avatar_url":"https://avatars.githubusercontent.com/u/755842?v=3","gravatar_id":"","url":"https://api.github.com/users/pokey","html_url":"https://github.com/pokey","followers_url":"https://api.github.com/users/pokey/followers","following_url":"https://api.github.com/users/pokey/following{/other_user}","gists_url":"https://api.github.com/users/pokey/gists{/gist_id}","starred_url":"https://api.github.com/users/pokey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pokey/subscriptions","organizations_url":"https://api.github.com/users/pokey/orgs","repos_url":"https://api.github.com/users/pokey/repos","events_url":"https://api.github.com/users/pokey/events{/privacy}","received_events_url":"https://api.github.com/users/pokey/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-14T14:50:48Z","updated_at":"2016-12-14T14:50:48Z","closed_at":null,"body":"When the tokenizer sees the [unicode apostrophe](http://www.fileformat.info/info/unicode/char/2019/index.htm), it doesn't tokenize correctly.  For example:\r\n\r\n```\r\nimport spacy\r\n\r\nnlp = spacy.load('en', parser=False)\r\n\r\nprint(list(nlp.tokenizer(\"I'm hungry\")))\r\nprint(list(nlp.tokenizer(\"I\\u2019m hungry\")))\r\n```\r\n\r\noutputs\r\n\r\n```\r\n[I, 'm, hungry]\r\n[I’m, hungry]\r\n```\r\n\r\n\r\n\r\n## My Environment\r\n* OS X 10.11.6\r\n* Python 3.5.2\r\n* spacy 1.2.0"}}
{"actor":{"display_login":"pokey"}}