<p>You can use embedding dropouts like this..</p>

<pre><code>with tf.variable_scope('embedding'):
   self.embedding_matrix = tf.get_variable( "embedding", shape=[self.vocab_size, self.embd_size], dtype=tf.float32, initializer=self.initializer)

with tf.name_scope("embedding_dropout"):
   self.embedding_matrix = tf.nn.dropout(self.embedding_matrix, keep_prob=self.embedding_dropout, noise_shape=[self.vocab_size,1])

with tf.name_scope('input'):
   self.input_batch = tf.placeholder(tf.int64, shape=(None, None))
   self.inputs = tf.nn.embedding_lookup(self.embedding_matrix, self.input_batch)
</code></pre>

<p>This randomly sets the rows of embedding matrix to zero, as mentioned in the <a href="https://arxiv.org/pdf/1512.05287.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1512.05287.pdf</a> which is cited in the paper you mentioned.</p>

<p>Source:</p>

<p><a href="https://github.com/tensorflow/tensorflow/issues/14746" rel="nofollow noreferrer">https://github.com/tensorflow/tensorflow/issues/14746</a></p>

<p>Similar pytorch implementation:</p>

<p><a href="https://github.com/salesforce/awd-lstm-lm/blob/master/embed_regularize.py" rel="nofollow noreferrer">https://github.com/salesforce/awd-lstm-lm/blob/master/embed_regularize.py</a></p>
Implement Embedding Dropout in Tensorflow
<p>I am reading this paper on "<a href="https://arxiv.org/pdf/1708.02182.pdf" rel="nofollow noreferrer">Regularizing and Optimizing LSTM Language Models</a>" and they talk about <code>Embedding Dropout</code> which says "As the dropout occurs on the embedding matrix that is used for a full forward and backward pass, this means that all occurrences of a specific word will disappear within that pass, equivalent to performing variational dropout on the connection between the one-hot embedding and the embedding lookup." However, I cannot seem to figure out a great approach to do this within a tensorflow experiment.  For each new batch, I currently embed my sequence with the following code:</p>

<pre><code>embedding_sequence = tf.contrib.layers.embed_sequence(features['input_sequence'], vocab_size=n_tokens, embed_dim=word_embedding_size)
</code></pre>

<p>Now I could easily apply dropout to the <code>embedding_sequence</code>, however my read of the paper says that the same words should be dropped from the entire forward/backward pass.  Any suggestions on a simple approach that would still allow me to use <code>embed_sequence</code>? Here is what I think my approach should be after breaking down <a href="https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/contrib/layers/python/layers/encoders.py#L91" rel="nofollow noreferrer">embed_sequence</a> but I'm still not convinced it is correct...</p>

<p><strong>PROPOSED SOLUTION</strong></p>

<pre><code>embedding_matrix = tf.get_variable("embeddings", shape=[vocab_size, embed_dim], dtype = tf.float32, initializer = None, trainable=True)
embedding_matrix_dropout = tf.nn.dropout(embedding_matrix, keep_prob=keep_prob)
embedding_sequence = tf.nn.embedding_lookup(embedding_matrix_dropout, features['input_sequence'])
</code></pre>

<p>Is there a more appropriate way to handle this? Is there anything I am getting from <code>embed_sequence</code> that I will not get from my proposed solution?</p>

<p>Secondary things I'm unsure about:</p>

<ol>
<li>what should my embedding_matrix initializer be? Default is set to None?</li>
<li><a href="https://www.tensorflow.org/api_docs/python/tf/nn/dropout" rel="nofollow noreferrer">tf.nn.dropout</a> appears to handle scaling by 1/keep_prob as mentioned is necessary in the paper, correct?</li>
</ol>