<p>As this question is rather open-ended I will start from the last parts, moving towards the more general answer to the main question posed in the title.</p>
<p><strong>Quick note:</strong> as pointed in the comments by <a href="https://stackoverflow.com/users/9094710/qusai-alothman">@Qusai Alothman</a>, you should find a better resource on the topic, this one is rather sparse when it comes to necessary informations.</p>
<p><strong>Additional note:</strong> full code for the process described in the last section would take <strong>way too much space</strong> to provide as an exact answer, it would be more of a blog post. I will highlight possible steps one should take to create such a network with helpful links as we go along.</p>
<p><strong>Final note:</strong> If there is anything dumb down there below (or you would like to expand the answer in any way or form, please do correct me/add info by posting a comment below).</p>
<h1>Question about the input</h1>
<p>Input here is generated from the <strong>random normal distribution</strong> and has <strong>no connection</strong> to the actual words. It is supposed to represent <a href="https://en.wikipedia.org/wiki/Word_embedding" rel="nofollow noreferrer">word embeddings</a>, e.g. representation of words as numbers carrying semantic (this is important!) meaning (sometimes depending on the context as well (see one of the current State Of The Art approaches, e.g. <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">BERT</a>)).</p>
<h2>Shape of the input</h2>
<p>In your example it is provided as:</p>
<p><code>seq_len, batch_size, embedding_size</code>,</p>
<p>where</p>
<ul>
<li><code>seq_len</code> - means length of a single sentence (varies across your
dataset), we will get to it later.</li>
<li><code>batch_size</code> - how many sentences
should be processed in one step of <code>forward</code> pass (in case of
<code>PyTorch</code> it is the forward method of class inheriting from
<a href="https://pytorch.org/docs/stable/nn.html#module" rel="nofollow noreferrer">torch.nn.Module</a>)</li>
<li><code>embedding_size</code> - vector with which one word is represented (it
might range from the usual <code>100/300</code> using <code>word2vec</code> up to <code>4096</code> or
so using the more recent approaches like the <strong>BERT</strong> mentioned
above)</li>
</ul>
<p>In this case it's all hard-coded of size one, which is not really useful for a newcomer, it only outlines the idea that way.</p>
<h3>Why does one need beginning of sentence and end of sentence tags in this case?</h3>
<p>Correct me if I'm wrong, but you don't need it <strong>if your input is separated into sentences</strong>. It is used if you provide <strong>multiple sentences</strong> to the model, and want to indicate <strong>unambiguously</strong> the beginning and end of each (used with models which depend on the previous/next sentences, it seems to not be the case here). Those are encoded by special tokens (the ones which are not present in the entire corpus), so neural network &quot;could learn&quot; they represent end and beginning of sentence (one special token for this approach would be enough).</p>
<p>If you were to use serious dataset, I would advise to split your text using libraries like <a href="https://spacy.io/" rel="nofollow noreferrer">spaCy</a> or <a href="https://www.nltk.org/" rel="nofollow noreferrer">nltk</a> (the first one is a pleasure to use IMO), they do a really good job for this task.</p>
<p>You dataset might be already splitted into sentences, in those cases you are kind of ready to go.</p>
<h3>Why don't I see the input being a corpus on which the model is trained like other classic NLP problems?</h3>
<p>I don't recall models being trained on the corpuses <strong>as is</strong>, e.g. using strings. Usually those are represented by floating-points numbers using:</p>
<ul>
<li>Simple approaches, e.g. <a href="https://en.wikipedia.org/wiki/Bag-of-words_model" rel="nofollow noreferrer">Bag Of
Words</a> or
<a href="https://pl.wikipedia.org/wiki/TFIDF" rel="nofollow noreferrer">TF-IDF</a></li>
<li>More sophisticated ones, which provide some information about word
relationships (e.g. <code>king</code> is more semantically related to <code>queen</code>
than to a, say, <code>banana</code>). Those were already linked above, some
other noticeable might be
<a href="https://nlp.stanford.edu/projects/glove/" rel="nofollow noreferrer">GloVe</a> or
<a href="https://arxiv.org/abs/1802.05365" rel="nofollow noreferrer">ELMo</a> and tons of other creative
approaches.</li>
</ul>
<h1>Question about the output</h1>
<p>One should output <strong>indices into <a href="https://pytorch.org/docs/stable/nn.html#embedding" rel="nofollow noreferrer">embeddings</a></strong>, which in turn correspond to words represented by a vector (more sophisticated approach mentioned above).</p>
<p>Each row in such embedding represents a unique word and it's respective columns are their unique representations (in PyTorch, first index might be reserved for the words for which a representation is unknown [if using pretrained embeddings], you may also delete those words, or represent them as aj average of sentence/document, there are some other viable approaches as well).</p>
<h2>Loss provided in the example</h2>
<pre><code># for language models, use cross-entropy :)
loss = nn.MSELoss()
</code></pre>
<p>For this task it makes no sense, as <a href="https://en.wikipedia.org/wiki/Mean_squared_error" rel="nofollow noreferrer">Mean Squared Error</a> is a regression metric, not a classification one.</p>
<p>We want to use one for classification, so <a href="https://en.wikipedia.org/wiki/Mean_squared_error" rel="nofollow noreferrer">softmax</a> should be used for multiclass case (we should be outputting numbers spanning <code>[0, N]</code>, where <code>N</code> is the number of unique words in our corpus).</p>
<p>PyTorch's <a href="https://en.wikipedia.org/wiki/Mean_squared_error" rel="nofollow noreferrer">CrossEntropyLoss</a> already takes logits (output of last layer <strong>without</strong> activation like softmax) and returns loss value for each example. I would advise this approach as it's numerically stable (and I like it as the most minimal one).</p>
<h1>I am trying to fill in the blank using a bidirectional RNN and pytorch</h1>
<p>This is a long one, I will only highlight steps I would undertake in order to create a model whose idea represents the one outlined in the post.</p>
<h2>Basic preparation of dataset</h2>
<p>You may use the one you mentioned above or start with something easier like <a href="https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html" rel="nofollow noreferrer">20 newsgroups</a> from <strong>scikit-learn</strong>.</p>
<p>First steps should be roughly this:</p>
<ul>
<li>scrape the metadata (if any) from your dataset (those might be HTML tags, some headers etc.)</li>
<li>split your text into sentences using a pre-made library (mentioned above)</li>
</ul>
<p>Next, you would like to create your target (e.g. words to be filled) in each sentence.
Each word should be replaced by a special token (say <code>&lt;target-token&gt;</code>) and moved to target.</p>
<p>Example:</p>
<ul>
<li>sentence: <code>Neural networks can do some stuff.</code></li>
</ul>
<p>would give us the following sentences and it's respective targets:</p>
<ul>
<li>sentence: <code>&lt;target-token&gt; networks can do some stuff.</code> target: <code>Neural</code></li>
<li>sentence: <code>Neural &lt;target-token&gt; can do some stuff.</code> target: <code>networks</code></li>
<li>sentence: <code>Neural networks &lt;target-token&gt; do some stuff.</code> target: <code>can</code></li>
<li>sentence: <code>Neural networks can &lt;target-token&gt; some stuff.</code> target: <code>do</code></li>
<li>sentence: <code>Neural networks can do &lt;target-token&gt; stuff.</code> target: <code>some</code></li>
<li>sentence: <code>Neural networks can do some &lt;target-token&gt;.</code> target: <code>some</code></li>
<li>sentence: <code>Neural networks can do some stuff &lt;target-token&gt;</code> target: <code>.</code></li>
</ul>
<p>You should adjust this approach to the problem at hand by correcting typos if there are any, tokenizing, lemmatizing and others, experiment!</p>
<h2>Embeddings</h2>
<p>Each word in each sentence should be replaced by an integer, which in turn points to it embedding.</p>
<p>I would advise you to use a pre-trained one. spaCy provides word vectors, but another interesting approach I would highly recommend is in the open source library <a href="https://github.com/zalandoresearch/flair" rel="nofollow noreferrer">flair</a>.</p>
<p>You may train your own, but it would take a lot of time + a lot of data for unsupervised training, and I think it is way beyond the scope of this question.</p>
<h2>Data batching</h2>
<p>One should use PyTorch's <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" rel="nofollow noreferrer">torch.utils.data.Dataset</a> and <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noreferrer">torch.utils.data.DataLoader</a>.</p>
<p>In my case, a good idea is was to provide custom <code>collate_fn</code> to <code>DataLoader</code>, which is responsible for creating padded batches of data (or represented as <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.PackedSequence" rel="nofollow noreferrer">torch.nn.utils.rnn.PackedSequence</a> already).</p>
<p><strong>Important:</strong> currently, you have to sort the batch by length (word-wise) and keep the indices able to &quot;unsort&quot; the batch into it's original form, you should remember that during implementation. You may use <code>torch.sort</code> for that task. In future versions of PyTorch, there is a chance, one might not have to do that, see <a href="https://github.com/pytorch/pytorch/issues/3584" rel="nofollow noreferrer">this issue</a>.</p>
<p>Oh, and remember to shuffle your dataset using <code>DataLoader</code>, while we're at it.</p>
<h2>Model</h2>
<p>You should create a proper model by inheriting from <code>torch.nn.Module</code>. I would advise you to create a more general model, where you can provide PyTorch's cells (like GRU, LSTM or RNN), multilayered and <strong>bidirectional</strong> (as is described in the post).</p>
<p>Something along those lines when it comes to model construction:</p>
<pre><code>import torch


class Filler(torch.nn.Module):
    def __init__(self, cell, embedding_words_count: int):
        self.cell = cell
        # We want to output vector of N
        self.linear = torch.nn.Linear(self.cell.hidden_size, embedding_words_count)

    def forward(self, batch):
        # Assuming batch was properly prepared before passing into the network
        output, _ = self.cell(batch)
        # Batch shape[0] is the length of longest already padded sequence
        # Batch shape[1] is the length of batch, e.g. 32
        # Here we create a view, which allows us to concatenate bidirectional layers in general manner
        output = output.view(
            batch.shape[0],
            batch.shape[1],
            2 if self.cell.bidirectional else 1,
            self.cell.hidden_size,
        )

        # Here outputs of bidirectional RNNs are summed, you may concatenate it
        # It makes up for an easier implementation, and is another often used approach
        summed_bidirectional_output = output.sum(dim=2)
        # Linear layer needs batch first, we have to permute it.
        # You may also try with batch_first=True in self.cell and prepare your batch that way
        # In such case no need to permute dimensions
        linear_input = summed_bidirectional_output.permute(1, 0, 2)
        return self.linear(embedding_words_count)
</code></pre>
<p>As you can see, information about shapes can be obtained in a general fashion. Such approach will allow you to create a model with how many layers you want, bidirectional or not (<code>batch_first</code> argument is problematic, but you can get around it too in a general way, left it out for improved clarity), see below:</p>
<pre><code>model = Filler(
    torch.nn.GRU(
        # Size of your embeddings, for BERT it could be 4096, for spaCy's word2vec 300
        input_size=300,
        hidden_size=100,
        num_layers=3,
        batch_first=False,
        dropout=0.4,
        bidirectional=True,
    ),
    # How many unique words are there in your dataset
    embedding_words_count=10000,
)
</code></pre>
<p>You may pass <code>torch.nn.Embedding</code> into your model (if pretrained and already filled), create it from numpy matrix or plethora of other approaches, <strong>it's highly dependent how your structure your code exactly</strong>. Still, please, make your code more general, <strong>do not hardcode shapes</strong> unless it's totally necessary (usually it's not).</p>
<p><strong>Remember it's only a showcase, you will have to tune and fix it on your own</strong>.
This implementation returns logits and no <code>softmax</code> layer is used. If you wish to calculate perplexity, you may have to add it in order to obtain a correct probability distribution across all possible vectors.</p>
<p>BTW: <a href="https://discuss.pytorch.org/t/concatenation-of-the-hidden-states-produced-by-a-bidirectional-lstm/3686" rel="nofollow noreferrer">Here</a> is some info on concatenation of bidirectional output of RNN.</p>
<h2>Model training</h2>
<p>I would <strong>highly recommend</strong> <a href="https://pytorch.org/ignite/" rel="nofollow noreferrer">PyTorch ignite</a> as it's quite customizable, you can log a lot of info using it, perform validation and abstract cluttering parts like for loops in training.</p>
<p>Oh, and split your model, training and others into separate modules, don't put everything into one unreadable file.</p>
<h1>Final notes</h1>
<p>This is the outline of how I would approach this problem, you may have more fun using attention networks instead of merely using the last output layer as in this example, though you shouldn't start with that.</p>
<p>And please check PyTorch's 1.0 documentation and <strong>do not</strong> follow blindly tutorials or blog posts you see online as they might be out of date really fast and quality of the code varies enormously. For example <a href="https://pytorch.org/docs/stable/autograd.html#variable-deprecated" rel="nofollow noreferrer">torch.autograd.Variable</a> is <strong>deprecated</strong> as can be seen in the link.</p>
How to fill in the blank using bidirectional RNN and pytorch?
<p>I am trying to fill in the blank using a bidirectional RNN and pytorch. </p>

<p>The input will be like: <code>The dog is _____, but we are happy he is okay.</code></p>

<p>The output will be like: </p>

<pre><code>1. hyper (Perplexity score here) 
2. sad (Perplexity score here) 
3. scared (Perplexity score here)
</code></pre>

<p>I discovered this idea here: <a href="https://medium.com/@plusepsilon/the-bidirectional-language-model-1f3961d1fb27" rel="noreferrer">https://medium.com/@plusepsilon/the-bidirectional-language-model-1f3961d1fb27</a></p>

<pre><code>import torch, torch.nn as nn
from torch.autograd import Variable

text = ['BOS', 'How', 'are', 'you', 'EOS']
seq_len = len(text)
batch_size = 1
embedding_size = 1
hidden_size = 1
output_size = 1

random_input = Variable(
    torch.FloatTensor(seq_len, batch_size, embedding_size).normal_(), requires_grad=False)

bi_rnn = torch.nn.RNN(
    input_size=embedding_size, hidden_size=hidden_size, num_layers=1, batch_first=False, bidirectional=True)

bi_output, bi_hidden = bi_rnn(random_input)

# stagger
forward_output, backward_output = bi_output[:-2, :, :hidden_size], bi_output[2:, :, hidden_size:]
staggered_output = torch.cat((forward_output, backward_output), dim=-1)

linear = nn.Linear(hidden_size * 2, output_size)

# only predict on words
labels = random_input[1:-1]

# for language models, use cross-entropy :)
loss = nn.MSELoss()
output = loss(linear(staggered_output), labels)
</code></pre>

<p>I am trying to reimplement the code above found at the bottom of the blog post. I am new to pytorch and nlp, and can't understand what the input and output to the code is.</p>

<p>Question about the input: I am guessing the input are the few words that are given. Why does one need beginning of sentence and end of sentence tags in this case? Why don't I see the input being a corpus on which the model is trained like other classic NLP problems? I would like to use the Enron email corpus to train the RNN.</p>

<p>Question about the output: I see the output is a tensor. My understanding is the tensor is a vector, so maybe a word vector in this case. How can you use the tensor to output the words themselves? </p>