title: Gensim train word2vec on wikipedia - preprocessing and parameters
body: <p>I am trying to train the word2vec model from <code>gensim</code> using the Italian wikipedia
  "<a href="http://dumps.wikimedia.org/itwiki/latest/itwiki-latest-pages-articles.xml.bz2" rel="nofollow noreferrer">http://dumps.wikimedia.org/itwiki/latest/itwiki-latest-pages-articles.xml.bz2</a>"</p>
  
  <p>However, I am not sure what is the best preprocessing for this corpus.</p>
  
  <p><code>gensim</code> model accepts a list of tokenized sentences.
  My first try is to just use the standard <code>WikipediaCorpus</code> preprocessor from <code>gensim</code>. This extract each article, remove punctuation and split words on spaces. With this tool each sentence would correspond to an entire model, and I am not sure of the impact of this fact on the model.</p>
  
  <p>After this I train the model with default parameters. Unfortunately after training it seems that I do not manage to obtain very meaningful similarities.</p>
  
  <p>What is the most appropriate preprocessing on the Wikipedia corpus for this task? (if this questions are too broad please help me by pointing to a relevant tutorial / article )</p>
  
  <p>This the code of my first trial:</p>
  
  <pre><code>from gensim.corpora import WikiCorpus
  import logging
  logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
  corpus = WikiCorpus('itwiki-latest-pages-articles.xml.bz2',dictionary=False)
  max_sentence = -1
  
  def generate_lines():
      for index, text in enumerate(corpus.get_texts()):
          if index &lt; max_sentence or max_sentence==-1:
              yield text
          else:
              break
  
  from gensim.models.word2vec import BrownCorpus, Word2Vec
  model = Word2Vec() 
  model.build_vocab(generate_lines()) #This strangely builds a vocab of "only" 747904 words which is &lt;&lt; than those reported in the literature 10M words
  model.train(generate_lines(),chunksize=500)
  </code></pre>