{"action":"created","issue":{"url":"https://api.github.com/repos/explosion/spaCy/issues/1752","repository_url":"https://api.github.com/repos/explosion/spaCy","labels_url":"https://api.github.com/repos/explosion/spaCy/issues/1752/labels{/name}","comments_url":"https://api.github.com/repos/explosion/spaCy/issues/1752/comments","events_url":"https://api.github.com/repos/explosion/spaCy/issues/1752/events","html_url":"https://github.com/explosion/spaCy/issues/1752","id":283751068,"number":1752,"title":"Multiple NER models in one pipeline","user":{"login":"tokestermw","id":4722119,"avatar_url":"https://avatars3.githubusercontent.com/u/4722119?v=4","gravatar_id":"","url":"https://api.github.com/users/tokestermw","html_url":"https://github.com/tokestermw","followers_url":"https://api.github.com/users/tokestermw/followers","following_url":"https://api.github.com/users/tokestermw/following{/other_user}","gists_url":"https://api.github.com/users/tokestermw/gists{/gist_id}","starred_url":"https://api.github.com/users/tokestermw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tokestermw/subscriptions","organizations_url":"https://api.github.com/users/tokestermw/orgs","repos_url":"https://api.github.com/users/tokestermw/repos","events_url":"https://api.github.com/users/tokestermw/events{/privacy}","received_events_url":"https://api.github.com/users/tokestermw/received_events","type":"User","site_admin":false},"labels":[{"id":111380485,"url":"https://api.github.com/repos/explosion/spaCy/labels/bug","name":"bug","color":"DD2A27","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-12-21T02:03:43Z","updated_at":"2018-03-27T06:47:09Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Hi, so let's say I've trained 3 different NER models from scratch: `ner_A`, `ner_B`, `ner_C`.\r\n\r\nI train using the `EntityRecognizer` pipeline with a different `.name` attribute.\r\n\r\nSince `EntityRecognizer` adds the `.ents` attributes to each `Doc` object, it doesn't seem like it's possible to add all the new NER models in one pipeline.\r\n\r\nA workaround is to save each NER model individually: `ner_A.to_disk('ner_A')`, `ner_B.to_disk('ner_B')`, `ner_C.to_disk('ner_C')`.\r\n\r\nFor production, we can load them back in individually: `ner_A.from_disk('ner_A', vocab=False)`, `ner_B.from_disk('ner_B', vocab=False)`, `ner_C.from_disk('ner_C', vocab=False)`.\r\n\r\nThen we can load the default nlp model: `nlp = spacy.load('en_core_web_sm')`. Run the original pipeline `doc = nlp('how are you?')`, then run the new NERs individually: `ner_A(doc)`, `ner_B(doc)`, `ner_C(doc)`.\r\n\r\nThen maybe add the results to custom attributes (e.g. `doc._.ner_A_ents`).\r\n\r\nWould this be a valid way to add models? I like using the spaCy sequence models since it has all the transition magic.\r\n\r\nThanks!\r\n\r\n## Your Environment\r\n<!-- Include details of your environment. If you're using spaCy 1.7+, you can also type \r\n`python -m spacy info --markdown` and copy-paste the result here.-->\r\n* **spaCy version:** 2.0.5\r\n* **Platform:** Darwin-17.3.0-x86_64-i386-64bit\r\n* **Python version:** 3.6.3\r\n* **Models:** en_core_web_sm"},"comment":{"url":"https://api.github.com/repos/explosion/spaCy/issues/comments/376415628","html_url":"https://github.com/explosion/spaCy/issues/1752#issuecomment-376415628","issue_url":"https://api.github.com/repos/explosion/spaCy/issues/1752","id":376415628,"user":{"login":"sudodoki","id":1506905,"avatar_url":"https://avatars3.githubusercontent.com/u/1506905?v=4","gravatar_id":"","url":"https://api.github.com/users/sudodoki","html_url":"https://github.com/sudodoki","followers_url":"https://api.github.com/users/sudodoki/followers","following_url":"https://api.github.com/users/sudodoki/following{/other_user}","gists_url":"https://api.github.com/users/sudodoki/gists{/gist_id}","starred_url":"https://api.github.com/users/sudodoki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sudodoki/subscriptions","organizations_url":"https://api.github.com/users/sudodoki/orgs","repos_url":"https://api.github.com/users/sudodoki/repos","events_url":"https://api.github.com/users/sudodoki/events{/privacy}","received_events_url":"https://api.github.com/users/sudodoki/received_events","type":"User","site_admin":false},"created_at":"2018-03-27T06:47:09Z","updated_at":"2018-03-27T06:47:09Z","author_association":"NONE","body":"can anyone on the thread who managed to find a workaround share snippet, please?\r\nI'm trying to use `en` model and `xx` for ner and running into similar issue:\r\n- if I do\r\n```\r\nnlp = en_core_web_md.load()\r\nner = xx_ent_wiki_sm.load()\r\n```\r\nand then do `doc = nlp(sent); ners = ner(sent)` I get\r\n```\r\n/usr/local/lib/python3.6/site-packages/thinc/neural/_classes/static_vectors.py in begin_update(self, ids, drop)\r\n     65                 sgd(self._mem.weights, self._mem.gradient, key=self.id)\r\n     66             return None\r\n---> 67         dotted = self.ops.batch_dot(vectors, self.W)\r\n     68         mask = self.ops.get_dropout_mask((dotted.shape[1],), drop)\r\n     69         if mask is not None:\r\n\r\nops.pyx in thinc.neural.ops.NumpyOps.batch_dot()\r\n\r\nValueError: shapes (15,0) and (300,128) not aligned: 0 (dim 1) != 300 (dim 0)\r\n```\r\n- if I do\r\n```\r\nnlp = spacy.load('en_core_web_md')\r\nner = spacy.load('xx', disable=['tokenizer', 'tagger', 'parser'])\r\nnlp.replace_pipe('ner', ner.pipeline[0][1])\r\n```\r\nI get the same error as above"}}
{"actor":{"display_login":"sudodoki"}}