<p>One simple way to go from word-vectors, to a single vector for a range-of-text, is to average the vectors together. And, that often works well-enough for some tasks. </p>

<p>However, that's not how the <code>Doc2Vec</code> class in <code>gensim</code> does it. That class implements the <a href="https://arxiv.org/abs/1405.4053" rel="noreferrer">'Paragraph Vectors' technique</a>, where separate document-vectors are trained in a manner analogous to word-vectors. </p>

<p>The doc-vectors participate in training a bit like a floating synthetic word, involved in every sliding window/target-word-prediction. They're <em>not</em> composed-up or concatenated-from preexisting word-vectors, though in some modes they may be simultaneously trained alongside word-vectors. (However, the fast and often top-performing PV-DBOW mode, enabled in gensim with the parameter <code>dm=0</code>, doesn't train or use input-word-vectors at all. It just trains doc-vectors that are good for predicting the words in each text-example.)</p>

<p>Since you've mentioned multiple libraries (both Spark MLib and gensim), but you've not shown your code, it's not certain exactly what <em>your</em> 
existing process is doing.</p>
How does Pyspark Calculate Doc2Vec from word2vec word embeddings?
<p>I have a pyspark dataframe with a corpus of ~300k unique rows each with a "doc" that contains a few sentences of text in each.</p>

<p>After processing, I have a 200 dimension vectorized representation of each row/doc. My NLP Process: </p>

<ol>
<li>Remove Punctuation with regex udf </li>
<li>Word Stemming with nltk snowball udf)</li>
<li>Pyspark Tokenizer</li>
<li>Word2Vec (ml.feature.Word2Vec, vectorSize=200, windowSize=5)</li>
</ol>

<p>I understand how this implementation uses the skipgram model to create embeddings for each word based on the full corpus used. My question is: <strong>How does this implementation go from a vector for each word in the corpus to a vector for each document/row?</strong></p>

<p>Is it the same processes as in the gensim doc2vec implementation where it simply concatenates the word vectors in each doc together?: <a href="https://stackoverflow.com/questions/40413866/how-does-gensim-calculate-doc2vec-paragraph-vectors">How does gensim calculate doc2vec paragraph vectors</a>. If so, how does it cut the vector down to the specified size of 200 (Does it use just the first 200 words? Average?)? </p>

<p>I was unable to find the information from the sourcecode: <a href="https://spark.apache.org/docs/2.2.0/api/python/_modules/pyspark/ml/feature.html#Word2Vec" rel="noreferrer">https://spark.apache.org/docs/2.2.0/api/python/_modules/pyspark/ml/feature.html#Word2Vec</a> </p>

<p>Any help or reference material to look at is super appreciated!</p>