{"action":"created","issue":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1261","repository_url":"https://api.github.com/repos/RaRe-Technologies/gensim","labels_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1261/labels{/name}","comments_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1261/comments","events_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1261/events","html_url":"https://github.com/RaRe-Technologies/gensim/issues/1261","id":219413246,"node_id":"MDU6SXNzdWUyMTk0MTMyNDY=","number":1261,"title":"Improve FastText loading times","user":{"login":"tmylk","id":1261528,"node_id":"MDQ6VXNlcjEyNjE1Mjg=","avatar_url":"https://avatars3.githubusercontent.com/u/1261528?v=4","gravatar_id":"","url":"https://api.github.com/users/tmylk","html_url":"https://github.com/tmylk","followers_url":"https://api.github.com/users/tmylk/followers","following_url":"https://api.github.com/users/tmylk/following{/other_user}","gists_url":"https://api.github.com/users/tmylk/gists{/gist_id}","starred_url":"https://api.github.com/users/tmylk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmylk/subscriptions","organizations_url":"https://api.github.com/users/tmylk/orgs","repos_url":"https://api.github.com/users/tmylk/repos","events_url":"https://api.github.com/users/tmylk/events{/privacy}","received_events_url":"https://api.github.com/users/tmylk/received_events","type":"User","site_admin":false},"labels":[{"id":234670,"node_id":"MDU6TGFiZWwyMzQ2NzA=","url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels/medium","name":"medium","color":"00d000","default":false},{"id":708430967,"node_id":"MDU6TGFiZWw3MDg0MzA5Njc=","url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels/perfomance","name":"perfomance","color":"d93f0b","default":false},{"id":175642,"node_id":"MDU6TGFiZWwxNzU2NDI=","url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels/wishlist","name":"wishlist","color":"d7e102","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2017-04-04T23:12:08Z","updated_at":"2018-07-30T10:00:50Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Consider reading just the `bin` file as sugested in https://github.com/RaRe-Technologies/gensim/issues/814#issuecomment-289464725\r\n\r\nCompare to C++ code in https://github.com/salestock/fastText.py/blob/77bdf69ee97a7446e314b342b129c5d46e9e4e29/fasttext/fasttext.pyx#L143"},"comment":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/comments/408812228","html_url":"https://github.com/RaRe-Technologies/gensim/issues/1261#issuecomment-408812228","issue_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1261","id":408812228,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODgxMjIyOA==","user":{"login":"menshikh-iv","id":15094669,"node_id":"MDQ6VXNlcjE1MDk0NjY5","avatar_url":"https://avatars2.githubusercontent.com/u/15094669?v=4","gravatar_id":"","url":"https://api.github.com/users/menshikh-iv","html_url":"https://github.com/menshikh-iv","followers_url":"https://api.github.com/users/menshikh-iv/followers","following_url":"https://api.github.com/users/menshikh-iv/following{/other_user}","gists_url":"https://api.github.com/users/menshikh-iv/gists{/gist_id}","starred_url":"https://api.github.com/users/menshikh-iv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/menshikh-iv/subscriptions","organizations_url":"https://api.github.com/users/menshikh-iv/orgs","repos_url":"https://api.github.com/users/menshikh-iv/repos","events_url":"https://api.github.com/users/menshikh-iv/events{/privacy}","received_events_url":"https://api.github.com/users/menshikh-iv/received_events","type":"User","site_admin":false},"created_at":"2018-07-30T10:00:49Z","updated_at":"2018-07-30T10:00:49Z","author_association":"MEMBER","body":"Thanks @DomHudson, I checked myself and confrim long time of `init_ngrams_post_load` (~90%).\r\nNoe: original python wrapper re-use c++ function for loading: https://github.com/facebookresearch/fastText/blob/master/python/fastText/pybind/fasttext_pybind.cc#L142\r\n\r\n\r\nProfiling:\r\n- `python2.7`\r\n- `gensim==3.5.0`\r\n- `wiki.en.bin` from https://fasttext.cc/docs/en/pretrained-vectors.html\r\n\r\n```python\r\n# content of t.py\r\nfrom gensim.models import FastText\r\nimport logging\r\n\r\n\r\nlogging.basicConfig(level=logging.INFO)\r\n\r\n\r\nm = FastText.load_fasttext_format(\"wiki.en.bin\")\r\n```\r\n\r\nrun it as `python -m cProfile -o p.prof t.py`\r\n\r\n\r\nresult in attach, `init_ngrams_post_load` definitely consume much time and this should be improved\r\n"}}
{"actor":{"display_login":"menshikh-iv"}}