{"action":"closed","issue":{"url":"https://api.github.com/repos/explosion/spaCy/issues/2682","repository_url":"https://api.github.com/repos/explosion/spaCy","labels_url":"https://api.github.com/repos/explosion/spaCy/issues/2682/labels{/name}","comments_url":"https://api.github.com/repos/explosion/spaCy/issues/2682/comments","events_url":"https://api.github.com/repos/explosion/spaCy/issues/2682/events","html_url":"https://github.com/explosion/spaCy/issues/2682","id":351651539,"node_id":"MDU6SXNzdWUzNTE2NTE1Mzk=","number":2682,"title":"Custom Tokenizer not loading from disk","user":{"login":"bellme","id":7198239,"node_id":"MDQ6VXNlcjcxOTgyMzk=","avatar_url":"https://avatars2.githubusercontent.com/u/7198239?v=4","gravatar_id":"","url":"https://api.github.com/users/bellme","html_url":"https://github.com/bellme","followers_url":"https://api.github.com/users/bellme/followers","following_url":"https://api.github.com/users/bellme/following{/other_user}","gists_url":"https://api.github.com/users/bellme/gists{/gist_id}","starred_url":"https://api.github.com/users/bellme/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bellme/subscriptions","organizations_url":"https://api.github.com/users/bellme/orgs","repos_url":"https://api.github.com/users/bellme/repos","events_url":"https://api.github.com/users/bellme/events{/privacy}","received_events_url":"https://api.github.com/users/bellme/received_events","type":"User","site_admin":false},"labels":[{"id":111380490,"node_id":"MDU6TGFiZWwxMTEzODA0OTA=","url":"https://api.github.com/repos/explosion/spaCy/labels/usage","name":"usage","color":"087EA6","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-08-17T16:05:10Z","updated_at":"2018-09-12T12:42:51Z","closed_at":"2018-09-12T12:42:51Z","author_association":"NONE","body":"I have the same issue as posted here https://github.com/explosion/spaCy/issues/2676.\r\n\r\nI am trying to package and load a custom spacy tokenizer using the following steps:\r\n\r\n1. modify custom tokenizer code\r\n```\r\nclass GranularWhitespaceTokenizer(Tokenizer):\r\n\r\n    def __init__(self, vocab):\r\n        super().__init__(vocab,\r\n                         rules={},\r\n                         prefix_search=prefix_re.search,\r\n                         suffix_search=suffix_re.search,\r\n                         infix_finditer=infix_re.finditer,\r\n                         token_match=simple_re.match)\r\n\r\n    def __call__(self, string, *args, **kwargs):\r\n        print('calling!')\r\n\r\n        return super().__call__(html.unescape(unescape(string)))\r\n```\r\n\r\nHere\r\n\r\n```\r\nnlp = spacy.blank('en', disable=['ner', 'parser', 'tagger'])\r\nnlp.tokenizer = GranularWhitespaceTokenizer(nlp.vocab)\r\nassert len(nlp(\"CamelCase\")) == 2\r\n\r\ndoc = u\"What's happened to me? he thought. &gt; &lt; &quot; It wasn't a dream.\"\r\ndoc = nlp.tokenizer(doc)\r\nprint([t.text for t in doc])\r\n```\r\nreturns\r\n```\r\ncalling!\r\n['What', \"'s\", 'happened', 'to', 'me', '?', 'he', 'thought', '.', '>', '<', '\"', 'It', 'wasn', \"'t\", 'a', 'dream', '.']\r\n```\r\n\r\n2. Save custom tokenizer to disk\r\n```\r\nLanguage.to_disk(nlp, \"granular_whitespace_tokenizer\")\r\n```\r\n3. Package tokenizer with spacy:\r\n```\r\n$spacy package /path/to/granular_whitespace_tokenizer /path/to/save/package/to\r\n```\r\n4. Install package\r\n```\r\n$ cd /path/to/save/to/en_granular_whitepace_tokenizer-0.0.0\r\n$ python setup.py sdist\r\n$ pip install .\r\n```\r\n5. Load saved model\r\n```\r\nimport spacy\r\nimport en_granular_whitespace_tokenizer\r\n\r\ndoc = u\"What's happened to me? he thought. &gt; &lt; &quot; It wasn't a dream.\"\r\n\r\nnlp = spacy.blank('en', disable=['ner', 'parser', 'tagger'])\r\nnlp.tokenizer = en_granular_whitespace_tokenizer.load()\r\n\r\ndoc = nlp.tokenizer(doc)\r\nassert len(nlp(\"CamelCase\")) == 2\r\nprint([t.text for t in doc])\r\n```\r\nreturns\r\n```\r\n['What', \"'s\", 'happened', 'to', 'me', '?', 'he', 'thought', '.', '&', 'gt', ';', '&', 'lt', ';', '&', 'quot', ';', 'It', 'was', \"n't\", 'a', 'dream', '.']\r\n```\r\n\r\nI'm confused as to why it doesn't seem to load the custom code properly. Can you point out with code examples what I'm missing here, either in this code or in additional specific __init__.py or other files? Your guidance is much appreciated, thank you!!\r\n\r\n## Your Environment\r\n<!-- Include details of your environment. If you're using spaCy 1.7+, you can also type \r\n`python -m spacy info --markdown` and copy-paste the result here.-->\r\n    spaCy version      2.0.12         \r\n    Location           /anaconda3/envs/nlp_label/lib/python3.6/site-packages/spacy\r\n    Platform           Darwin-17.7.0-x86_64-i386-64bit\r\n    Python version     3.6.6          \r\n    Models  \r\n\r\n"}}
{"actor":{"display_login":"ines"}}