Surprisingly this wasn't asked before - at least I didn't find anything besides some vaguely related questions.

So, what is a recurrent neural network, and what are their advantages over regular NNs?

  • 2
    $\begingroup$ In the 1990s Mark W. Tilden has introduced the first BEAM robotics walker. The system is based on the nv-neuron which is an oscillating neural network. Tilden has called the concept bicores, but it's the same like a recurrent neural network. Explaining the inner working in a few sentences is a bit complicated. The more easier way to introduce the technology is an autonomous boolean network. This logic gate network contains of a feedback loop which means the system is oscillating. In contrast to a boolean logic gate, a recurrent neural network has more features and can be trained by algorithms. $\endgroup$ – Manuel Rodriguez Apr 28 at 18:59
  • 1
    $\begingroup$ this blog post has an awesome explanation: colah.github.io/posts/2015-08-Understanding-LSTMs $\endgroup$ – MrE May 24 at 17:46

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information.

Difference with traditional Neural networks using pictures from this book:

enter image description here

And, an RNN:

enter image description here

Notice the difference -- feedforward neural networks' connections do not form cycles. If we relax this condition, and allow cyclical connections as well, we obtain recurrent neural networks (RNNs). You can see that in the hidden layer of the architecture.

While the difference between a multilayer perceptron and an RNN may seem trivial, the implications for sequence learning are far-reaching. An MLP can only map from input to output vectors, whereas an RNN can in principle map from the entire history of previous inputs to each output. Indeed, the equivalent result to the universal approximation theory for MLPs is that an RNN with a sufficient number of hidden units can approximate any measurable sequence-to-sequence mapping to arbitrary accuracy.

Important takeaway:

The recurrent connections allow a 'memory' of previous inputs to persist in the network's internal state, and thereby influence the network output.

Talking in terms of advantages is not appropriate as they both are state-of-the-art and are particularly good at certain tasks. A broad category of tasks that RNN excel at is:

Sequence Labelling

The goal of sequence labelling is to assign sequences of labels, drawn from a fixed alphabet, to sequences of input data.

Ex: Transcribe a sequence of acoustic features with spoken words (speech recognition), or a sequence of video frames with hand gestures (gesture recognition).

Some of the sub-tasks in sequence labelling are:

Sequence Classification

Label sequences are constrained to be of length one. This is referred to as sequence classification, since each input sequence is assigned to a single class. Examples of sequence classification task include the identification of a single spoken work and the recognition of an individual handwritten letter.

Segment Classification

Segment classification refers to those tasks where the target sequences consist of multiple labels, but the locations of the labels -- that is, the positions of the input segments to which the labels apply -- are known in advance.

  • $\begingroup$ very nice answer thanks! I am starting to regret not taking that Systems and Control theory class. Seems like useful stuff, feedback loops and all that, to know in the context of NNs. $\endgroup$ – olinarr Apr 29 at 7:19
  • 1
    $\begingroup$ Welcome! They certainly are useful. $\endgroup$ – naive Apr 29 at 7:27

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks.

An RNN can be trained using back-propagation through time (BBTT), such that these backward or self-connections "memorise" previously seen inputs. Hence, these connections are mainly used to track temporal relations between elements of a sequence of inputs, which makes RNNs well suited to sequence prediction and similar tasks.

There are several RNN models: for example, RNNs with LSTM or GRU units. LSTM (or GRU) is an RNN whose single units perform a more complex transformation than a unit in a "plain RNN", which performs a linear transformation of the input followed by the application of a non-linear function (e.g. ReLU) to this linear transformation. In theory, "plain RNN" are as powerful as RNNs with LSTM units. In practice, they suffer from the "vanishing and exploding gradients" problem. Hence, in practice, LSTMs (or similar sophisticated recurrent units) are used.


Your Answer

