{"action":"created","issue":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1035","repository_url":"https://api.github.com/repos/RaRe-Technologies/gensim","labels_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1035/labels{/name}","comments_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1035/comments","events_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1035/events","html_url":"https://github.com/RaRe-Technologies/gensim/issues/1035","id":193300973,"number":1035,"title":"Gensim Word2Vec model Export in TSV for Google's Embedding Projector Visualizer","user":{"login":"loretoparisi","id":163333,"avatar_url":"https://avatars.githubusercontent.com/u/163333?v=3","gravatar_id":"","url":"https://api.github.com/users/loretoparisi","html_url":"https://github.com/loretoparisi","followers_url":"https://api.github.com/users/loretoparisi/followers","following_url":"https://api.github.com/users/loretoparisi/following{/other_user}","gists_url":"https://api.github.com/users/loretoparisi/gists{/gist_id}","starred_url":"https://api.github.com/users/loretoparisi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/loretoparisi/subscriptions","organizations_url":"https://api.github.com/users/loretoparisi/orgs","repos_url":"https://api.github.com/users/loretoparisi/repos","events_url":"https://api.github.com/users/loretoparisi/events{/privacy}","received_events_url":"https://api.github.com/users/loretoparisi/received_events","type":"User","site_admin":false},"labels":[{"id":175640,"url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels/feature","name":"feature","color":"0b02e1","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-12-03T17:02:23Z","updated_at":"2016-12-05T15:46:56Z","closed_at":null,"body":"I'm using gensym [Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html) module to build a linguistic model of text documents.\r\nI've recently found the the Google's Embedding Projector described in [here](https://www.tensorflow.org/versions/master/how_tos/embedding_viz/index.html) and that looks like the attachment depicted here showing as an example of the word embedding [PCA](http://setosa.io/ev/principal-component-analysis/) visualization. The paper behind this visualization is [here](https://arxiv.org/pdf/1611.05469.pdf).\r\n\r\n![schermata 2016-12-03 alle 15 36 06](https://cloud.githubusercontent.com/assets/163333/20859993/3d12a9ba-b96e-11e6-83d5-134d835266bc.jpg)\r\n\r\nI'm trying to export to word embedding in the TSV file format needed to represent t-SNE and PCA, that is M lines of a 200 (or N) features vector columns of real numbers in this specific case, but I'm not sure hot to export my model data in this format.\r\n\r\nMy training python script that builds the model looks like the following, where I'm using a custom iterator, since I'm loading data from documents in streaming mode:\r\n\r\n```python\r\n\r\n# gensim Logging code taken from http://rare-technologies.com/word2vec-tutorial/\r\n    logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\r\n\r\n    logger.info( \"Training corpora [%s] language:%s...\" % (corpora_filepath, corpora_language) )\r\n\r\n    # My Corpus\r\n    # word deaccent, sentences tokenize, include stopwords, max 100 corpora files\r\n    corpus_iterator = TextCorpus( corpora_filepath, deaccent=True, tokenize=True, stopwords=False, maxFiles=100 )\r\n\r\n    # min_count = ignore all words with total frequency lower than this.\r\n    words_min_count = 1\r\n    # number of features\r\n    # window is the maximum distance between the current and predicted word within a sentence.\r\n    features_size = 300\r\n    # window size\r\n    # window is the maximum distance between the current and predicted word within a sentence.\r\n    words_distance_window = 4\r\n    #  if 1 (default), sort the vocabulary by descending frequency before assigning word indexes.\r\n    sorted_vocab = 1\r\n    # sg defines the training algorithm. By default (sg=0), CBOW is used. Otherwise (sg=1), skip-gram is employed.\r\n    train_algorithm = 1\r\n    \r\n    #negative = if > 0, negative sampling will be used, the int for negative specifies how many “noise words” should be drawn (usually between 5-20). \r\n    # Default is 5. If set to 0, no negative samping is used.\r\n    negative_sampling = 5\r\n\r\n    #  limit RAM during vocabulary building; if there are more unique words than this, then prune the infrequent ones. \r\n    # Every 10 million word types need about 1GB of RAM. Set to None for no limit (default).\r\n    max_vocab_memory_size = None\r\n\r\n# for other parameters\r\n    # @see https://radimrehurek.com/gensim/models/word2vec.html\r\n    model = Word2Vec(corpus_iterator, \r\n        sg = train_algorithm, \r\n        min_count=words_min_count, \r\n        size=features_size, \r\n        window=words_distance_window, \r\n        sorted_vocab=sorted_vocab,\r\n        negative=negative_sampling,\r\n        max_vocab_size=max_vocab_memory_size)\r\n\r\n    #igram = Phrases( tokens )\r\n\r\n    #print \"\\bigram samples %s is %d\" % (bigram, len(bigram))\r\n\r\n    #print \"Training 2gram...\"\r\n    #model.train( bigram[ tokens ] )\r\n\r\n    # save the model\r\n\r\n    model_name_prefix='TextDoc'\r\n\r\n    # model saved in the gensym word2vec format\r\n    model_name = model_filepath+model_name_prefix+'-'+corpora_language\r\n    \r\n    logger.info( \"Saving trained model...\" )\r\n    model.save( model_name )\r\n    logger.info( \"Model saved to %s\" % (model_name) )\r\n\r\n    # this model is a binary file in the original word2vec format\r\n    model_name = model_filepath+model_name_prefix+'-vectors-negative'+str(features_size)+'-'+corpora_language+'.bin'\r\n    \r\n    # the vocabulary is a text file order by frequency desc\r\n    # wordN frequency\r\n    # wordN-1 frequency\r\n    vocab_name = model_filepath+model_name_prefix+'-vocab-vectors-negative'+str(features_size)+'-'+corpora_language+'.txt'\r\n    \r\n    logger.info( \"Saving trained model and vocabulary word2vec format...\" )\r\n    model.save_word2vec_format(model_name, fvocab=vocab_name, binary=True)\r\n    logger.info( \"Binary model saved to %s vocabulary %s\" % (model_name, vocab_name) )\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/comments/264888816","html_url":"https://github.com/RaRe-Technologies/gensim/issues/1035#issuecomment-264888816","issue_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/1035","id":264888816,"user":{"login":"loretoparisi","id":163333,"avatar_url":"https://avatars.githubusercontent.com/u/163333?v=3","gravatar_id":"","url":"https://api.github.com/users/loretoparisi","html_url":"https://github.com/loretoparisi","followers_url":"https://api.github.com/users/loretoparisi/followers","following_url":"https://api.github.com/users/loretoparisi/following{/other_user}","gists_url":"https://api.github.com/users/loretoparisi/gists{/gist_id}","starred_url":"https://api.github.com/users/loretoparisi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/loretoparisi/subscriptions","organizations_url":"https://api.github.com/users/loretoparisi/orgs","repos_url":"https://api.github.com/users/loretoparisi/repos","events_url":"https://api.github.com/users/loretoparisi/events{/privacy}","received_events_url":"https://api.github.com/users/loretoparisi/received_events","type":"User","site_admin":false},"created_at":"2016-12-05T15:46:56Z","updated_at":"2016-12-05T15:46:56Z","body":"@tmylk thanks the idea could be to have a `save`api that writes down this `TVS`file like `model.save_word2vec_tsv_format`. I'm pretty sure that this could be done as @piskvorky described above and I will try something like that, but I'm not so deep inside gensym at this time to know how do to that, but I will try something and update here.\r\n\r\n"}}
{"actor":{"display_login":"loretoparisi"}}