title: does word2vec tutorial example imply potential sub-optimal implementation?
body: <p><strong>Machine Specs:</strong></p>
  
  <ul>
  <li>Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz</li>
  <li>16GB RAM</li>
  <li>nvidia TitanX</li>
  </ul>
  
  <p>(btw
  There is a bug in word2vec example(both word2vec.py and word2vec_optimized.py) that it would run into error if no device directive is given. I add a with tf.device('/cpu:0') to work around)  </p>
  
  <p><strong>Hyper-Parameter Setup:</strong></p>
  
  <ul>
  <li>the default model hyper-parameter + 8 concurrent threads</li>
  <li>use text8 as training data and questions-words.txt as test.</li>
  </ul>
  
  <p><strong>Profiling Result:</strong></p>
  
  <ul>
  <li><p>word2vec_optimized.py: 171891 words/sec ( 8 threads)</p></li>
  <li><p>word2vec_optimized.py: 9258 words/sec (8 threads)</p></li>
  </ul>
  
  <p>(the optimized version is comparable with Mikolov's original implementation, and the standard one is significantly slower)</p>
  
  <p>CPU utilization during training is roughly the same (the optimized version is a little bit higher)</p>
  
  <p>I reviewed the code of two the implementations, and I couldn't figure out why the standard implementation (word2vec.py) is 20x slower than the C++ hand-coded kernel(word2vec_kernels.cc)</p>
  
  <p>The word2vec_kernel.cc does not seem to do anything special to speed up (it first do a batch random sampling and then run a loop over samples in a single batch)</p>
  
  <p>The word2vec.py version doesn't seem to be sub-optimal to me either, it uses the standard tensorflow operations and the batching seems to be suitable.</p>
  
  <p>Have you guys ever do any internal profiling to figure out what is the performance bottleneck of the standard version? Would the performance bug be a threat to the training performance of other neural networks in general?</p>