<blockquote>
  <p>If I have to use pretrained word vectors as embedding layer in Neural
  Networks (eg. say CNN), How do I deal with index 0?</p>
</blockquote>

<p><strong>Answer</strong></p>

<p>In general, empty entries can be handled via a weighted cost of the model and the targets.
However, when dealing with words and sequential data, things can be a little tricky and there are several things that can be considered. Let's make some assumptions and work with that.</p>

<p><strong>Assumptions</strong></p>

<ol>
<li>We begin with a pre-trained word2vec model.</li>
<li>We have sequences with varying lengths, with at most <code>max_lenght</code> words.</li>
</ol>

<p><strong>Details</strong></p>

<ul>
<li>Word2Vec is a model that learns a mapping (embedding) from discrete variables (word token = word unique id) to a continuous vector space.</li>
<li>The representation in the vector space is such that the cost function (CBOW, Skip-gram, essentially it is predicting word from context in bi-directional way) is minimized on the corpus. </li>
<li>Reading basic tutorials (like <a href="https://www.tensorflow.org/tutorials/representation/word2vec" rel="nofollow noreferrer">Google's word2vec tutorial</a> on <a href="https://www.tensorflow.org/tutorials/" rel="nofollow noreferrer">Tensorflow tutorials</a>) reveals some details on the algorithm, including <a href="https://en.wikipedia.org/wiki/Word2vec" rel="nofollow noreferrer">negative sampling</a>.</li>
<li>The implementation is a lookup table. It is faster than the alternative one-hot encoding technique, since the dimensions of a one-hot encoded matrix are huge (say 10,000 columns for 10,000 words, <code>n</code> row for <code>n</code> sequential words). So the lookup (hash) table is significantly faster, and it selects rows from the embedding matrix (for row vectors).</li>
</ul>

<p><strong>Task</strong></p>

<ul>
<li>Add missing entries (no words) and use it in the model.</li>
</ul>

<p><strong>Suggestions</strong></p>

<ul>
<li>If there is some use for the cost of missing data, such as using a prediction from that entry and there is a label for that entry, you can add a new value as suggested (can be the 0 index, but all indexes must move <code>i=i+1</code> and the embedding matrix should have new row at position 0).</li>
<li>Following the first suggestion, you need to train the added row. You can use negative sampling for the NaN class vs all. <em>I do not suggest it for handling missing values. It is a good trick to handle an "Unknown word" class.</em></li>
<li>You can weight the cost of those entries by constant 0 for each sample that is shorter that <code>max_length</code>. That is, if we have a sequence of word tokens <code>[0,5,6,2,178,24,0,NaN,NaN]</code>, the corresponding weight vector is <code>[1,1,1,1,1,1,1,0,0]</code></li>
<li>You should worry about re-indexing the words and the cost of it. In memory, there is almost no difference (<code>1</code> vs <code>N</code> words, <code>N</code> is large). In complexity, it is something that can be later incorporated in the initial tokenize function. The predictions and model complexity is a larger issue and more important requirement from the system.</li>
<li>There are numerous ways to tackle varying lengths (LSTM, RNNs, now we try CNNs and costs tricks). Read state-of-the-art literature on that issue, I'm sure there is much work. For example, see <a href="https://arxiv.org/abs/1404.2188" rel="nofollow noreferrer">A Convolutional Neural Network for Modelling Sentences</a> paper.</li>
</ul>
Pretrained (Word2Vec) embedding in Neural Networks
<p>If I have to use pretrained word vectors as embedding layer in Neural Networks (eg. say CNN), How do I deal with index 0?</p>

<p><strong>Detail:</strong> </p>

<p>We usually start with creating a zero numpy 2D array. Later we fill in the indices of words from the vocabulary. 
The problem is, 0 is already the index of another word in our vocabulary (say, 'i' is index at 0). Hence, we are basically initializing the whole matrix filled with 'i' instead of empty words. So, how do we deal with padding all the sentences of equal length?</p>

<p>One easy pop-up in mind is we can use the another digit=numberOfWordsInVocab+1 to pad. But wouldn't that take more size? [Help me!]</p>