title: Gensim vector shape changing
body: <p>i am trying to vectorize a text and classify it by using gensim and tensorflow.keras.</p>
  <p>Before the train I have shapes as follows:</p>
  <pre><code>
  X_train, y_train (1019471, 100, 1) (1019471, 5) 
  X_validate, y_validate (127419, 100, 1) (127419, 5) 
  X_test, y_test (127476, 100, 1) (127476, 5)
  
  </code></pre>
  <pre><code>    for function in functionSource:
          func_list.append([str(preprocess_text(function))])
      return func_list
  </code></pre>
  <pre><code>def preprocess_text(sen):
      # Remove punctuations and numbers
      sentence = re.sub('[^a-zA-Z]', ' ', sen)
  
      # Single character removal
      sentence = re.sub(r&quot;\s+[a-zA-Z]\s+&quot;, ' ', sentence)
  
      # Removing multiple spaces
      sentence = re.sub(r'\s+', ' ', sentence)
  
      return sentence
  </code></pre>
  <pre><code>    embedding_dim = 100
      model = Word2Vec(func_ist, min_count=1,size= embedding_dim, workers=6, window =3, sg = 1)
  </code></pre>
  <p>After the train I have the following shapes:</p>
  <pre><code>X_train, y_train (1018222, 100, 1) (1019471, 5) 
  X_validate, y_validate (127398, 100, 1) (127419, 5) 
  X_test, y_test (127461, 100, 1) (127476, 5)
  </code></pre>
  <p><strong>logging for gensim</strong></p>
  <pre><code>INFO: 2020-07-20 12:44:54,936 - word2vec.py:1596 - scan_vocab : collected 1018222 word types from a corpus of 1019471 raw words and 1019471 sentences
  INFO: 2020-07-20 12:44:54,936 - word2vec.py:1647 - prepare_vocab : Loading a fresh vocabulary
  INFO: 2020-07-20 12:44:56,790 - word2vec.py:1671 - prepare_vocab : effective_min_count=1 retains 1018222 unique words (100% of original 1018222, drops 0)
  INFO: 2020-07-20 12:44:56,790 - word2vec.py:1677 - prepare_vocab : effective_min_count=1 leaves 1019471 word corpus (100% of original 1019471, drops 0)
  INFO: 2020-07-20 12:44:58,912 - word2vec.py:1736 - prepare_vocab : deleting the raw counts dictionary of 1018222 items
  INFO: 2020-07-20 12:44:58,930 - word2vec.py:1739 - prepare_vocab : sample=0.001 downsamples 0 most-common words
  INFO: 2020-07-20 12:44:58,930 - word2vec.py:1742 - prepare_vocab : downsampling leaves estimated 1019471 word corpus (100.0% of prior 1019471)
  INFO: 2020-07-20 12:45:00,543 - base_any2vec.py:1022 - estimate_memory : estimated required memory for 1018222 words and 100 dimensions: 1323688600 bytes
  INFO: 2020-07-20 12:45:00,543 - word2vec.py:1888 - reset_weights : resetting layer weights
  </code></pre>
  <p>I couldn't understand the part that my dataset losing some of the text, can anyone explain to me the reason why the shape gets smaller?</p>
  <p>i am trying to learn about the topic. i would be grateful if anyone can explain it</p>