{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1820","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1820/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1820/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1820/events","html_url":"https://github.com/tensorflow/tensorflow/issues/1820","id":146865416,"number":1820,"title":"Tensorflow version of word2vec issue with huge corpora","user":{"login":"JosuGoiko","id":15345919,"avatar_url":"https://avatars.githubusercontent.com/u/15345919?v=3","gravatar_id":"","url":"https://api.github.com/users/JosuGoiko","html_url":"https://github.com/JosuGoiko","followers_url":"https://api.github.com/users/JosuGoiko/followers","following_url":"https://api.github.com/users/JosuGoiko/following{/other_user}","gists_url":"https://api.github.com/users/JosuGoiko/gists{/gist_id}","starred_url":"https://api.github.com/users/JosuGoiko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JosuGoiko/subscriptions","organizations_url":"https://api.github.com/users/JosuGoiko/orgs","repos_url":"https://api.github.com/users/JosuGoiko/repos","events_url":"https://api.github.com/users/JosuGoiko/events{/privacy}","received_events_url":"https://api.github.com/users/JosuGoiko/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/contributions%20welcome","name":"contributions welcome","color":"009800"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-04-08T08:55:59Z","updated_at":"2016-07-21T12:11:45Z","closed_at":null,"body":" Hi All,\r\n\r\nI have the pip wheel distribution of Tensorflow r7.1 in Red Hat 6 in our cluster.\r\n\r\nMy issue is that I wanted to extract the embedding from a huge corpora (5*10⁹ tokens), but  the 32 bit \"corpus_size_\" counter in \"word2vec_kernel.cc\" is not enough. It seems that it is not so straightforward to fix it (changing the counter to int64). \r\n\r\nI have seen that this very error was reported last  17th of Dec 2015 in issue #531.\r\n\r\nI also receive the following error:\r\n\r\nW tensorflow/models/embedding/word2vec_kernels.cc:59] Invalid argument: The text file wbu.txt contains too little data: -522037214 words\r\nE tensorflow/core/framework/op_segment.cc:53] Create kernel failed: Invalid argument: The text file wbu.txt contains too little data: -522037214 words\r\nE tensorflow/core/common_runtime/executor.cc:275] Executor failed to create kernel. Invalid argument: The text file wbu.txt contains too little data: -522037214 words\r\n     [[Node: Skipgram = Skipgram[batch_size=16, filename=\"wbu.txt\", min_count=5, subsample=0.001, window_size=5, _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\r\nTraceback (most recent call last):\r\n  File \"word2vec_minibatch.py\", line 425, in <module>\r\n    tf.app.run()\r\n  File \"/opt/python-2.7/lib/python2.7/site-packages/tensorflow/python/platform/default/_app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"word2vec_minibatch.py\", line 411, in main\r\n    model = Word2Vec(opts, session)\r\n  File \"word2vec_minibatch.py\", line 175, in __init__\r\n    self.build_graph()\r\n  File \"word2vec_minibatch.py\", line 312, in build_graph\r\n    opts.words_per_epoch) = self._session.run([words, counts, words_per_epoch])\r\n  File \"/opt/python-2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 315, in run\r\n    return self._run(None, fetches, feed_dict)\r\n  File \"/opt/python-2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 511, in _run\r\n    feed_dict_string)\r\n  File \"/opt/python-2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 564, in _do_run\r\n    target_list)\r\n  File \"/opt/python-2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 586, in _do_call\r\n    e.code)\r\ntensorflow.python.framework.errors.InvalidArgumentError: The text file wbu.txt contains too little data: -522037214 words\r\n     [[Node: Skipgram = Skipgram[batch_size=16, filename=\"wbu.txt\", min_count=5, subsample=0.001, window_size=5, _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\r\nCaused by op u'Skipgram', defined at:\r\n  File \"word2vec_minibatch.py\", line 425, in <module>\r\n    tf.app.run()\r\n  File \"/opt/python-2.7/lib/python2.7/site-packages/tensorflow/python/platform/default/_app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"word2vec_minibatch.py\", line 411, in main\r\n    model = Word2Vec(opts, session)\r\n\r\nThanx\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/234235451","html_url":"https://github.com/tensorflow/tensorflow/issues/1820#issuecomment-234235451","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/1820","id":234235451,"user":{"login":"Robin-des-Bois","id":6424472,"avatar_url":"https://avatars.githubusercontent.com/u/6424472?v=3","gravatar_id":"","url":"https://api.github.com/users/Robin-des-Bois","html_url":"https://github.com/Robin-des-Bois","followers_url":"https://api.github.com/users/Robin-des-Bois/followers","following_url":"https://api.github.com/users/Robin-des-Bois/following{/other_user}","gists_url":"https://api.github.com/users/Robin-des-Bois/gists{/gist_id}","starred_url":"https://api.github.com/users/Robin-des-Bois/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Robin-des-Bois/subscriptions","organizations_url":"https://api.github.com/users/Robin-des-Bois/orgs","repos_url":"https://api.github.com/users/Robin-des-Bois/repos","events_url":"https://api.github.com/users/Robin-des-Bois/events{/privacy}","received_events_url":"https://api.github.com/users/Robin-des-Bois/received_events","type":"User","site_admin":false},"created_at":"2016-07-21T12:11:44Z","updated_at":"2016-07-21T12:11:44Z","body":"Maybe this helps someone: \r\nFor me, It still does not work on OS X using clang (7.3.0) but it works on CentOS 6 using GCC (4.8.2 and 4.9.2)."}}
{"actor":{"display_login":"Robin-des-Bois"}}