title: Dumping spark word2vec vectors to a file
body: <p>I am using spark mllib to generate word vectors. I wish to fit all my data and then get the trained word vectors and dump them to a file.</p>
  
  <p>I am doing this :</p>
  
  <pre><code>JavaRDD&lt;List&lt;String&gt;&gt; data = javaSparkContext.parallelize(streamingData, partitions);
  Word2Vec word2vec = new Word2Vec();
  Word2VecModel model = word2vec.fit(data);
  </code></pre>
  
  <p>So, if my training data had sentences like </p>
  
  <pre><code>I love Spark
  </code></pre>
  
  <p>I want to save the output in files as :</p>
  
  <pre><code>I       0.03 0.53 0.12...
  love    0.31 0.14 0.12...
  Spark   0.41 0.18 0.84...
  </code></pre>
  
  <p>After training, I am getting the vectors from the model object like this</p>
  
  <pre><code>Map&lt;String, float[]&gt; wordMap = JavaConverters.mapAsJavaMapConverter(model.getVectors()).asJava();
  List&lt;String&gt; wordvectorlist = Lists.newArrayList();
  for (String s : wordMap.keySet()) {
      StringBuilder wordvector = new StringBuilder(s);
      for (float f : wordMap.get(s)) {
          wordvector.append(" " + f);
      }
      wordvectorlist.add(wordvector.toString());
      if (wordvectorlist.size() &gt; 1000000) {
          writeToFile(wordvectorlist);
          wordvectorlist.clear();
      }
  
  }
  </code></pre>
  
  <p>I will be generating these word vectors for a very huge data (~1.5 TB) and thus, I might not be able to save the returned object Word2VecModel in memory of my driver.
  How can I store this wordvectors map as a rdd so that I can write to files without storing the full map in driver memory?</p>
  
  <p>I looked into word2vec implementation of deeplearning4j but that implementation also requires loading all the vectors in driver memory.</p>