{"action":"created","issue":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2378","repository_url":"https://api.github.com/repos/RaRe-Technologies/gensim","labels_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2378/labels{/name}","comments_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2378/comments","events_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2378/events","html_url":"https://github.com/RaRe-Technologies/gensim/issues/2378","id":408428670,"node_id":"MDU6SXNzdWU0MDg0Mjg2NzA=","number":2378,"title":"loading fastText model trained with pretrained_vectors still fails (see: #2350)","user":{"login":"cbjrobertson","id":22371870,"node_id":"MDQ6VXNlcjIyMzcxODcw","avatar_url":"https://avatars1.githubusercontent.com/u/22371870?v=4","gravatar_id":"","url":"https://api.github.com/users/cbjrobertson","html_url":"https://github.com/cbjrobertson","followers_url":"https://api.github.com/users/cbjrobertson/followers","following_url":"https://api.github.com/users/cbjrobertson/following{/other_user}","gists_url":"https://api.github.com/users/cbjrobertson/gists{/gist_id}","starred_url":"https://api.github.com/users/cbjrobertson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cbjrobertson/subscriptions","organizations_url":"https://api.github.com/users/cbjrobertson/orgs","repos_url":"https://api.github.com/users/cbjrobertson/repos","events_url":"https://api.github.com/users/cbjrobertson/events{/privacy}","received_events_url":"https://api.github.com/users/cbjrobertson/received_events","type":"User","site_admin":false},"labels":[{"id":175641,"node_id":"MDU6TGFiZWwxNzU2NDE=","url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels/bug","name":"bug","color":"e10c02","default":true},{"id":1162250977,"node_id":"MDU6TGFiZWwxMTYyMjUwOTc3","url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels/fasttext","name":"fasttext","color":"ad52ea","default":false},{"id":708355863,"node_id":"MDU6TGFiZWw3MDgzNTU4NjM=","url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels/need%20info","name":"need info","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"mpenkov","id":1101396,"node_id":"MDQ6VXNlcjExMDEzOTY=","avatar_url":"https://avatars2.githubusercontent.com/u/1101396?v=4","gravatar_id":"","url":"https://api.github.com/users/mpenkov","html_url":"https://github.com/mpenkov","followers_url":"https://api.github.com/users/mpenkov/followers","following_url":"https://api.github.com/users/mpenkov/following{/other_user}","gists_url":"https://api.github.com/users/mpenkov/gists{/gist_id}","starred_url":"https://api.github.com/users/mpenkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpenkov/subscriptions","organizations_url":"https://api.github.com/users/mpenkov/orgs","repos_url":"https://api.github.com/users/mpenkov/repos","events_url":"https://api.github.com/users/mpenkov/events{/privacy}","received_events_url":"https://api.github.com/users/mpenkov/received_events","type":"User","site_admin":false},"assignees":[{"login":"mpenkov","id":1101396,"node_id":"MDQ6VXNlcjExMDEzOTY=","avatar_url":"https://avatars2.githubusercontent.com/u/1101396?v=4","gravatar_id":"","url":"https://api.github.com/users/mpenkov","html_url":"https://github.com/mpenkov","followers_url":"https://api.github.com/users/mpenkov/followers","following_url":"https://api.github.com/users/mpenkov/following{/other_user}","gists_url":"https://api.github.com/users/mpenkov/gists{/gist_id}","starred_url":"https://api.github.com/users/mpenkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpenkov/subscriptions","organizations_url":"https://api.github.com/users/mpenkov/orgs","repos_url":"https://api.github.com/users/mpenkov/repos","events_url":"https://api.github.com/users/mpenkov/events{/privacy}","received_events_url":"https://api.github.com/users/mpenkov/received_events","type":"User","site_admin":false}],"milestone":null,"comments":13,"created_at":"2019-02-09T12:49:09Z","updated_at":"2019-02-20T13:50:05Z","closed_at":null,"author_association":"NONE","body":"## Description\r\nLoading pretrained `fastext_model.bin` with `gensim.models.fasttext.FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin')` fails with `AssertionError: unexpected number of vectors` despite fix for #2350.\r\n\r\n## Steps/Code/Corpus to Reproduce\r\nfirst install `develop` branch with: `pip install --upgrade git+git://github.com/RaRe-Technologies/gensim@develop`, then:\r\n```python\r\n#dependencies \r\nimport requests, zipfile, io\r\nfrom gensim.models.fasttext import FastText\r\n\r\n#download model\r\nft_url = 'https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki-news-300d-1M-subword.bin.zip'\r\nr = requests.get(ft_url)\r\nz = zipfile.ZipFile(io.BytesIO(r.content))\r\nz.extractall()\r\n\r\n#attempt load\r\nmod = FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin')\r\n```\r\n\r\n#### Expected Results\r\nLoaded model.\r\n\r\n#### Actual Results\r\n```\r\n---------------------------------------------------------------------------\r\nAssertionError                            Traceback (most recent call last)\r\n<ipython-input-29-a054256d6f88> in <module>\r\n      1 #load model\r\n      2 from gensim.models.fasttext import FastText\r\n----> 3 mod = FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin')\r\n      4 # from gensim.models import KeyedVectors\r\n      5 # wv = KeyedVectors.load_word2vec_format('wiki-news-300d-1M-subword.vec')\r\n\r\n/anaconda3/envs/tensor_env/lib/python3.7/site-packages/gensim/models/fasttext.py in load_fasttext_format(cls, model_file, encoding, full_model)\r\n   1012 \r\n   1013         \"\"\"\r\n-> 1014         return _load_fasttext_format(model_file, encoding=encoding, full_model=full_model)\r\n   1015 \r\n   1016     def load_binary_data(self, encoding='utf8'):\r\n\r\n/anaconda3/envs/tensor_env/lib/python3.7/site-packages/gensim/models/fasttext.py in _load_fasttext_format(model_file, encoding, full_model)\r\n   1270     #\r\n   1271     # We explicitly set min_count=1 regardless of the model's parameters to\r\n-> 1272     # ignore the trim rule when building the vocabulary.  We do this in order\r\n   1273     # to support loading native models that were trained with pretrained vectors.\r\n   1274     # Such models will contain vectors for _all_ encountered words, not only\r\n\r\n/anaconda3/envs/tensor_env/lib/python3.7/site-packages/gensim/models/keyedvectors.py in init_post_load(self, vectors, match_gensim)\r\n   2205         \"\"\"\r\n   2206         vocab_words = len(self.vocab)\r\n-> 2207         assert vectors.shape[0] == vocab_words + self.bucket, 'unexpected number of vectors'\r\n   2208         assert vectors.shape[1] == self.vector_size, 'unexpected vector dimensionality'\r\n   2209 \r\n\r\nAssertionError: unexpected number of vectors\r\n```\r\n\r\n## Versions\r\nDarwin-18.2.0-x86_64-i386-64bit\r\nPython 3.7.2 (default, Dec 29 2018, 00:00:04) \r\n[Clang 4.0.1 (tags/RELEASE_401/final)]\r\nNumPy 1.16.1\r\nSciPy 1.2.0\r\ngensim 3.7.1\r\nFAST_VERSION 1\r\n\r\n#### thanks for your work! \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/comments/465581480","html_url":"https://github.com/RaRe-Technologies/gensim/issues/2378#issuecomment-465581480","issue_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2378","id":465581480,"node_id":"MDEyOklzc3VlQ29tbWVudDQ2NTU4MTQ4MA==","user":{"login":"cbjrobertson","id":22371870,"node_id":"MDQ6VXNlcjIyMzcxODcw","avatar_url":"https://avatars1.githubusercontent.com/u/22371870?v=4","gravatar_id":"","url":"https://api.github.com/users/cbjrobertson","html_url":"https://github.com/cbjrobertson","followers_url":"https://api.github.com/users/cbjrobertson/followers","following_url":"https://api.github.com/users/cbjrobertson/following{/other_user}","gists_url":"https://api.github.com/users/cbjrobertson/gists{/gist_id}","starred_url":"https://api.github.com/users/cbjrobertson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cbjrobertson/subscriptions","organizations_url":"https://api.github.com/users/cbjrobertson/orgs","repos_url":"https://api.github.com/users/cbjrobertson/repos","events_url":"https://api.github.com/users/cbjrobertson/events{/privacy}","received_events_url":"https://api.github.com/users/cbjrobertson/received_events","type":"User","site_admin":false},"created_at":"2019-02-20T13:50:05Z","updated_at":"2019-02-20T13:50:05Z","author_association":"NONE","body":"Yes.\n\nCole\n\n\nOn Wed, Feb 20, 2019 at 1:27 PM Michael Penkov <notifications@github.com>\nwrote:\n\n> That commit is irrelevant to the current issue.\n>\n> So, you can load the model correctly using the unicode branch?\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/RaRe-Technologies/gensim/issues/2378#issuecomment-465573040>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AVVeHqQJqqx-583I8bFHharUHiuKFBuQks5vPU1DgaJpZM4ayhus>\n> .\n>\n"}}
{"actor":{"display_login":"cbjrobertson"}}