AI News, Recurrent Neural Networks Tutorial, Part 1 &#8211; Introduction to RNNs



It&#8217;s a multi-part series in which I&#8217;m planning to cover the following: As part of the tutorial we will implement a recurrent neural network based language model.

The applications of language models are two-fold: First, it allows us to score arbitrary sentences based on how likely they are to occur in the real world.


By unrolling we simply mean that we write out the network for the complete sequence. For example, if the sequence we care about is a sentence of 5 words, the network would be unrolled into a 5-layer neural network, one layer for each word. The formulas that govern the computation happening in a RNN are as follows: There are a few things to note here: RNNs have shown great success in many NLP tasks.

A side-effect of being able to predict the next word is that we get a generative model, which allows us to generate new text by sampling from the output probabilities.

In Language Modeling our input is typically a sequence of words (encoded as one-hot vectors for example), and our output is the sequence of predicted words.

A key difference is that our output only starts after we have seen the complete input, because the first word of our translated sentences may require information captured from the complete input sequence.

Research papers about Machine Translation: Given an input sequence of acoustic signals from a sound wave, we can predict a sequence of phonetic segments together with their probabilities.


The memory in LSTMs are called cells and you can think of them as black boxes that take as input the previous state and current input .

I hope you&#8217;ve gotten a basic understanding of what RNNs are and what they can do. In the next post we&#8217;ll implement a first version of our language model RNN using Python and Theano. Please leave questions in the comments!

The Unreasonable Effectiveness of Recurrent Neural Networks




































Neural Machine Translation (seq2seq) Tutorial

Authors: Thang Luong, Eugene Brevdo, Rui Zhao (Google Research Blogpost, Github) This version of the tutorial requires TensorFlow Nightly. For

using the stable TensorFlow versions, please consider other branches such as tf-1.4.


of seq2seq models and shows how to build a competitive seq2seq model

We achieve this goal by: We believe that it is important to provide benchmarks that people can easily replicate.

As a result, we have provided full experimental results and pretrained

on models on the following publicly available datasets: We first build up some basic knowledge about seq2seq models for NMT, explaining how

and tricks to build the best possible NMT models (both in speed and translation

RNNs, beam search, as well as scaling up to multiple GPUs using GNMT attention.

Back in the old days, traditional phrase-based translation systems performed their

task by breaking up source sentences into multiple chunks and then translated

An encoder converts a source sentence into a 'meaning' vector which is passed


manner, NMT addresses the local translation problem in the traditional phrase-based

differ in terms of: (a) directionality – unidirectional or bidirectional;

In this tutorial, we consider as examples a deep multi-layer RNN which is unidirectional

simply consumes the input source words without making any prediction;

on the other hand, processes the target sentence while predicting the next

running: Let's first dive into the heart of building an NMT model with concrete code snippets

At the bottom layer, the encoder and decoder RNNs receive as input the following:

are in time-major format and contain word indices: Here for efficiency, we train with multiple sentences (batch_size) at once.

The embedding weights, one set per language, are usually

choose to initialize embedding weights with pretrained word representations such

Once retrieved, the word embeddings are then fed as input into the main network, which

consists of two multi-layer RNNs – an encoder for the source language and a

models do a better job when fitting large training datasets).

RNN uses zero vectors as its starting states and is built as follows: Note that sentences have different lengths to avoid wasting computation, we tell dynamic_rnn

describe how to build multi-layer LSTMs, add dropout, and use attention in a

Given the logits above, we are now ready to compute our training loss: Here, target_weights is a zero-one matrix of the same size as decoder_outputs.

SGD with a learning of 1.0, the latter approach effectively uses a much smaller

pass is just a matter of a few lines of code: One of the important steps in training RNNs is gradient clipping.

a decreasing learning rate schedule, which yields better performance.

nmt/scripts/ /tmp/nmt_data Run the following command to start the training: The above command trains a 2-layer LSTM seq2seq model with 128-dim hidden units and

We can start Tensorboard to view the summary of the model during training: Training the reverse direction from English and Vietnamese can be done simply by changing:\

--src=en --tgt=vi While you're training your NMT models (and once you have trained models), you can

obtain translations given previously unseen source sentences.

Greedy decoding – example of how a trained NMT model produces a translation

for a source sentence 'Je suis étudiant' using greedy search.

the correct target words as an input, inference uses words predicted by the

know the target sequence lengths in advance, we use maximum_iterations to limit

Having trained a model, we can now create an inference file and translate some sentences:

Remember that in the vanilla seq2seq model, we pass the last source state from the

It consists of the following stages: Here, the function score is used to compared the target hidden state \( h_t \) with

each of the source hidden states \( \overline{h}_s \), and the result is normalized to produced

\( a_t \) is used to derive the softmax logit and loss.

and on whether the previous state \( h_{t-1} \) is used instead of \(

h_t \) in the scoring function as originally suggested in (Bahdanau et al., 2015).

of attention, i.e., direct connections between target and source, needs to be

use the current target hidden state as a 'query' to decide on which parts of

mechanism, we happen to use the set of source hidden states (or their transformed

versions, e.g., \( W_1h_t \) in Bahdanau's scoring style) as 'keys'.

Thanks to the attention wrapper, extending our vanilla seq2seq code with attention First, we need to define an attention mechanism, e.g., from (Luong et al., 2015):

to create a new directory for the attention model, so we don't reuse the previously

Run the following command to start the training: After training, we can use the same inference command with the new out_dir for inference:

separate graphs: Building separate graphs has several benefits: The primary source of complexity becomes how to share Variables across the three graphs

Before: Three models in a single graph and sharing a single Session After: Three models in three graphs, with three Sessions sharing the same Variables Notice how the latter approach is 'ready' to be converted to a distributed version.

feed data at each call (and thereby performing our own batching,

training and eval pipelines: The first approach is easier for users who aren't familiar with TensorFlow or need

to do exotic input modification (i.e., their own minibatch queueing) that can

Some examples: All datasets can be treated similarly via input processing.

To convert each sentence into vectors of word strings, for example, we use the dataset

map transformation: We can then switch each sentence vector into a tuple containing both the vector and

object table, this map converts the first tuple elements from a vector of strings

containing the tuples of the zipped lines can be created via: Batching of variable-length sentences is straightforward.

Values emitted from this dataset will be nested tuples whose tensors have a leftmost

The structure will be: Finally, bucketing that batches similarly-sized source sentences together is also

Reading data from a Dataset requires three lines of code: create the iterator, get

Bidirectionality on the encoder side generally gives better performance (with some

of how to build an encoder with a single bidirectional layer: The variables encoder_outputs and encoder_state can be used in the same way as

While greedy decoding can give us quite reasonable translation quality, a beam search

explore the search space of all possible translations by keeping around a small

a minimal beam width of, say size 10, is generally sufficient.

You may notice the speed improvement of the attention based NMT model is very small

(i.e., 1 bidirectional layers for the encoder), embedding dim is

measure the translation quality in terms of BLEU scores (Papineni et al., 2002).

step-time means the time taken to run one mini-batch (of size 128).

(i.e., 2 bidirectional layers for the encoder), embedding dim is

These results show that our code builds strong baseline systems for NMT.\ (Note

that WMT systems generally utilize a huge amount monolingual data which we currently do not.) Training Speed: (2.1s step-time, 3.4K wps) on Nvidia K40m

see the speed-ups with GNMT attention, we benchmark on K40m only: These results show that without GNMT attention, the gains from using multiple gpus are minimal.\ With

The above results show our models are very competitive among models of similar architectures.\ [Note

that OpenNMT uses smaller models and the current best result (as of this writing) is 28.4 obtained by the Transformer network (Vaswani et al., 2017) which has a significantly different architecture.] We have provided a

There's a wide variety of tools for building seq2seq models, so we pick one per language:\ Stanford [PyTorch] We would like to thank Denny Britz, Anna Goldie, Derek Murray, and Cinjon Resnick for their work bringing new features to TensorFlow and the seq2seq library.

Recurrent Neural Networks (RNNs) : Part 1

There is a vast amount of data which is inherently sequential, such as speech, time series (weather, financial, etc.), sensor data, video, and text, just to mention some.

Even though it does not seem to be the most exciting task in the world on the surface, this type of modelling is an essential building block for understanding natural language and a fundamental task in natural language processing (NLP).

Being this article intended to be an introduction to Natural Language Understanding and text generation, we relegate most of the practical and implementation details for the next post of the series in which we will take a more hands-on approach to text generation in different forms.

We can now start formalizing our ideas, let's consider a sentence $S$ composed by $T$ words, such that $$S = (w_1 , w _2 , ...

, w _T)$$ At the same time, each symbol $w_i$ is part of a vocabulary $V$ which contains all the possible words, $$V = \{ v_1, v _2, ..., v _{|V|} \}$$ where $|V|$ represents the size of the vocabulary.

language model, in general, attempts to predict the next word $w_{t+1}$ given the preceding words $w _{\lt t}$ at each time step $t$.

Now we have an architecture which can receive different inputs at each time step $x_t$, has the capability of producing outputs at each time step $o_t$ and maintains a memory state $h_t$ which contains information about what happened in the network up to time $t$.

For now, we can think of the RNN cell as a computation in which we update the memory vector $h$ deciding, at each timestep, which information we want to keep, which information is not relevant anymore and we would like to forget and which information to add from the new input.

$$ w _i = [0,0,...,1 (i\text{-th element}), 0,...,0]^T \in \{ 0, 1\}^{|V|} $$ The prior knowledge embedded in this encoding scheme is minimal in the sense that the distance between any two word vectors is equal to 1 if the two words are different and 0 if the words are the same word.

Each of these vectors will be then multiplied by a weight matrix $\mathbf{E}$, to get a sequence of continuous vectors $(\mathbf{x} _1, \mathbf{x} _2, ..., \mathbf{x} _{T-1})$ such that, $$ \mathbf{x} _j = \mathbf{E}^T \mathbf{w} _j $$ Performance note: Actually this matrix-vector multiplication is not performed.

Since the vector $\mathbf{w} _j$ has only one element equal to 1 ($i$-th element) and the rest are zeros, the multiplication is equivalent to just taking the $i$-th row of $\mathbf{E}$.

Now we repeat the procedure for time step 1 in which And is the input of the cell, $h_1$ is the memory state which contains information about the past and $p(w_2|\text{&lt;\s> And})$ is the output.

The output layer of the RNN is then a softmax layer which returns a vector of size $|V|$ whose $i$-th element indicates the predicted probability of the word $V_i$ being the next word to appear in the sentence.

For the case in which the output at time $t$ is an affine transformation of the computed memory state $h_t$, we have $$ p(w_t = k |

w_{\lt t}) = \frac{exp(\mathbf{v} _k^T h _t + b _k)}{\sum _{k'} exp(\mathbf{v} _{k'}^T h _t + b _{k'})}$$ We have already an architecture that can potentially learn to score sequences making use of recurrent neural networks.

Using the chain rule and the fact that the log of a product equals the sum of the logs, we obtain, for a sentence $\mathbf{x}$, $$ L(\mathbf{x}) = - \sum_t log \ p _{model} (w _t = x _{t + 1}) = - \sum _t log \ \mathbf{o} _t [x _{t+1}] $$ where $\mathbf{o} _t [x _{t+1}]$ is the element of the output softmax corresponding to the real word $x _{t+1}$.

\times (n-1)} \rightarrow \mathbb R^d $$ The second stage, characterized by $g$ maps the continuous vector $h$ to the target word probability, by applying an affine transformation (multiplication by a matrix and addition to a bias vector) followed by a softmax normalization (to convert the output into a valid probability distribution).

Now, if we have two sequences of context words which are usually followed by a similar set of words, then the context vectors $\mathbf{h} _1$ and $\mathbf{h} _2$ have to be similar.

The target word vectors $\mathbf{u} _{teams}$ and $\mathbf{u} _{groups}$ will necessarily be close to each other as well, because otherwise the probability of 'teams' given 'four' ($p(\text{teams}|\text{four}) \propto \mathbf{u} _{teams} \mathbf{h} _{four}$) and the probability of 'groups' given 'four' ($p(\text{groups}|\text{four}) \propto \mathbf{u} _{groups} \mathbf{h} _{four}$) will be different despite the fact that they are equally likely in the training corpus (second and third sentences of the training set).

From this context vector, the model will have to assign a high probability to the word 'groups', because the context vector $\mathbf{h} _{three}$ and the target word vector $\mathbf{u} _{gropus}$ are well aligned.

On the one hand, it learns the probability of one word given the context, and in the other, it represents similar context by similar vectors in the context space and similar words by similar vectors in the word space.

If we want to generate a new sentence we just need to initialize the context vector $\mathbf{h} _0$ randomly, then unroll the RNN sampling at each time step one word from the output word probability distribution and feeding this word back to the input of the next time RNN unit.

Then, once our model has a certain understanding of words, context and text structure, we can continue with the training procedure just using Sheldon's lines to learn the particular way he speaks.

Creating A Language Translation Model Using Sequence To Sequence Learning Approach

Anyway, the toughest time has gone, and now I can get myself back to work, to bring to you guys new interesting (and maybe boring as usual) blog posts on Deep Learning.

What RNNs differ from normal Neural Networks is, instead of computing the output prediction on each input independently, RNNs compute the output of timestep \(t\) using not only the input of timestep \(t\), but also involving the input of previous timesteps (say, timestep \(t-1\), \(t-2\), \(\dots\)).

As you already saw in my previous post, inputs are actually sequences of characters, and each output was simply the corresponding input shifted by one character to the right.

If you haven’t read my previous post yet, please take a look at it by the link below (make sure you do before moving on): But here comes a big question: What if input sequence and output sequence have different lengths?

And in terms of Natural Language Processing (or NLP for short), you are more likely to face problems where their lengths are totally different, not only between each pair of input and output sequence, but also between input sequences themselves!

For example, in building a language translation model, each pair of input and output sequence are in different languages, so there’s a big chance that they don’t have the same length.

To make the problem become more concrete, let’s take a look at the graph below: (Image cut from the original paper of Sequence to Sequence Learning with Neural Networks) As illustrated in the graph above, we have “ABC” as the input sequence, and “WXYZ” as the output sequence.

As the names of the two networks are somehow self-explained, first, it’s clear that we can’t directly compute the output sequence by using just one network, so we need to use the first network to encode the input sequence into some kind of “middle sequence”, then the other network will decode that sequence into our desire output sequence.

Concretely, what the Encoder actually did is creating a temporary output vector from the input sequence (you can think about that temporary output vector as a sequence with only one timestep).

After repeating the output vector from the Encoder \(n\) times, we obtain a sequence with exact the same length with the associated output sequence, we can leave the computation for the Decoder network!

Here I prepared five sentences (they were actually from a great song of Twenty One Pilots, link provided at Reference) and let’s imagine that they will be the input sequences to our network.

To make them all equal in length, let’s take the length of the longest sentence as the common length, and we only need to add one same word some times to the end of the other two, until they have the same length as the longest one.

That’s the reason why I decided not to dig into details in the previous section, but to explain it along with the corresponding part in the code instead so that you won’t find it difficult to understand the abstract terms (at least I think so).

mapping the sentence a, b, c to the sentence α, β, γ, the LSTM is asked to map c, b, a to α, β, γ, where

As computers can only understand the gray scale values of pixels in an image, inputting sequences of raw human-alike words will make no sense to computers.

In real deep learning projects, especially when we’re dealing with NLP problems, our training data is pretty large in size, which the number of vocabularies may be up to millions.

So, what we do first is to count the frequency which a word appears in the text, then we create the vocabulary set using only 10000 words with highest frequencies (you can change to 20000 or more, but make sure that your machine can handle it).

With the vocabulary set we created above, it’s pretty easy to create an array to store only the words, and eliminate their frequencies of occurrence (we don’t need that information after all).

But you may wonder, we were supposed to create some kind of dictionary here in order to convert each index to its associated word, and now what I told you to create is an array.

Well, since we want to create the index-to-word dictionary, and we can access any element of an array through its index, it’s better just to create a simple array instead of a dictionary where keys are all indexes!

As I mentioned earlier, we need a word called ZERO in order to make all sequences have the exact same length, and another word called UNK, which stands for unknown words or out of vocabulary in order to represent words which are not in the vocabulary set.

And also remember that we’re only putting 10000 words with highest frequencies into the vocabulary set, which also means that our network will actually learn words from that vocabulary set only.

If we don’t, and have some word with index \(0\) instead, then our network won’t be able to decide whether that \(0\) is padded zero, or index of a particular word.

Concretely, we have to do a final processing step called vectorization: Explaining the process of vectorization (especially in terms of NLP) is kind of tedious, so I think it’s better help you guys have a visualization of it.

Since we need to compute only a single vector from the input sequences, the encoder network is pretty simple, just a network with a single hidden layer is far from enough.

In fact, we are supposed to input directly the vectorized array from above step into some kind of recurrent neural network like LSTM or vanilla RNN.

Due to some limitations of memory, I was able to train 1000 sequences, which means 1 batch at a time (with batch size 1000).

If you guys have some ideas about it, please kindly let me know :) At the time of writing, the model is on its third day of learning and everything seems promising.

For that reason, I don’t expect you to fully understand the idea behind it just by reading this blog post (I myself can’t say that I fully understand it, either!).

Lecture 8: Recurrent Neural Networks and Language Models

Lecture 8 covers traditional language models, RNNs, and RNN language models. Also reviewed are important training problems and tricks, RNNs for other sequence tasks, and bidirectional and deep...

Recurrent Neural Networks - Ep. 9 (Deep Learning SIMPLIFIED)


|Carsten van Weelden, Beata Nyari | Siamese LSTM in Keras: Learning Character-Based Phrase

PyData Amsterdam 2017 Siamese LSTM in Keras: Learning Character-Based Phrase Representations In this talk we will explain how we solved the problem of classifying job titles into a job ontology...


TensorFlow Training - ) This Edureka Recurrent Neural Networks tutorial video (Blog: will help you in understanding.

Lecture 13: Convolutional Neural Networks

Lecture 13 provides a mini tutorial on Azure and GPUs followed by research highlight "Character-Aware Neural Language Models." Also covered are CNN Variant 1 and 2 as well as comparison between...

Actionable and Political Text Classification Using Word Embeddings and LSTM

Author: Adithya Rao, Klout, Inc. Abstract: In this work, we apply word embeddings and neural networks with Long Short-Term Memory (LSTM) to text classification problems, where the classification...

Deep Learning: Using TensorFlow LSTM seq2seq Neural Nets to build Pronunciation System! (tPronounce)

I hope you enjoyed this tutorial! If you did, please make sure to leave a like, comment, and subscribe! It really does help out a lot! Contact: Email: Twitter: @TajyMany...

LSTM input output shape , Ways to improve accuracy of predictions in Keras

In this tutorial we look at how we decide the input shape and output shape for an LSTM. We also tweak various parameters like Normalization, Activation and the loss function and see their...

Recurrent Neural Networks (RNN / LSTM )with Keras - Python

In this tutorial, we learn about Recurrent Neural Networks (LSTM and RNN). Recurrent neural Networks or RNNs have been very successful and popular in time series data predictions. There are...

What are Recurrent Neural Networks (RNN) and Long Short Term Memory Networks (LSTM) ?

Recurrent Neural Networks or RNN have been very popular and effective with time series data. In this tutorial, we learn about RNNs, the Vanishing Gradient problem and the solution to the problem...