Vanishing gradient problem

In machine learning, the vanishing gradient problem is encountered when training artificial neural networks with gradient-based learning methods and backpropagation. In such methods, during each iteration of training each of the neural network's weights receives an update proportional to the partial derivative of the error function with respect to the current weight. The problem is that in some cases, the gradient will be vanishingly small, effectively preventing the weight from changing its value. In the worst case, this may completely stop the neural network from further training. As one example of the problem cause, traditional activation functions such as the hyperbolic tangent function have gradients in the range , and backpropagation computes gradients by the chain rule. This has the effect of multiplying of these small numbers to compute gradients of the early layers in an -layer network, meaning that the gradient (error signal) decreases exponentially with while the early layers train very slowly.
Back-propagation allowed researchers to train supervised deep artificial neural networks from scratch, initially with little success. Hochreiter's diplom thesis of 1991 formally identified the reason for this failure in the "vanishing gradient problem", which not only affects many-layered feedforward networks, but also recurrent networks. The latter are trained by unfolding them into very deep feedforward networks, where a new layer is created for each time step of an input sequence processed by the network. (The combination of unfolding and backpropagation is termed backpropagation through time.)
When activation functions are used whose derivatives can take on larger values, one risks encountering the related exploding gradient problem.
Prototypical models.
This section is based on.
Recurrent network model.
A generic recurrent network has hidden states formula_1 inputs formula_2, and outputs formula_3. Let it be parametrized by formula_4, so that the system evolves asformula_5Often, the output formula_6 is a function of formula_7, as some formula_8. The vanishing gradient problem already presents itself clearly when formula_9, so we simplify our notation to the special case with:formula_10 Now, take its differential:formula_11Training the network requires us to define a loss function to be minimized. Let it be formula_12(x_{t}, u_1, ..., u_t)&lt;/math&gt;
 for which the problem is the same, just with more complex notations.|name=let it be L|content=content|text=text}}, then minimizing it by gradient descent givesformula_13where formula_14 is the learning rate.
The vanishing/exploding gradient problem appears because there are repeated multiplications, of the formformula_15
Example: recurrent network with sigmoid activation.
For a concrete example, consider a typical recurrent network defined by
formula_16where formula_17 is the network parameter, formula_18 is the sigmoid activation function, applied to each vector coordinate separately, and formula_19 is the bias vector.
Then, formula_20, and so formula_21Since formula_22, the operator norm of the above multiplication is bounded above by formula_23. So if the spectral radius of formula_24 is formula_25, then at large formula_26, the above multiplication has operator norm bounded above by formula_27. This is the prototypical vanishing gradient problem.
The effect of a vanishing gradient is that the network cannot learn long-range effects. Recall Equation ():formula_28The components of formula_29 are just components of formula_30 and formula_31, so if formula_32 are bounded, then formula_33 is also bounded by some formula_34, and so the terms in formula_35 decay as formula_36. This means that, effectively, formula_35 is affected only by the first formula_38 terms in the sum.
If formula_39, the above analysis does not quite work. For the prototypical exploding gradient problem, the next model is clearer.
Dynamical systems model.
Following (Doya, 1993), consider this one-neuron recurrent network with sigmoid activation:formula_40At the small formula_41 limit, the dynamics of the network becomesformula_42Consider first the autonomous case, with formula_43. Set formula_44, and vary formula_19 in formula_46. As formula_19 decreases, the system has 1 stable point, then has 2 stable points and 1 unstable point, and finally has 1 stable point again. Explicitly, the stable points are formula_48.
Now consider formula_49 and formula_50, where formula_51 is large enough that the system has settled into one of the stable points.
If formula_52 puts the system very close to an unstable point, then a tiny variation in formula_53 or formula_19 would make formula_55 move from one stable point to the other. This makes formula_49 and formula_50 both very large, a case of the exploding gradient.
If formula_52 puts the system far from an unstable point, then a small variation in formula_53 would have no effect on formula_55, making formula_61, a case of the vanishing gradient.
Note that in this case, formula_62 neither decays to zero nor blows up to infinity. Indeed, it's the only well-behaved gradient, which explains why early researches focused on learning or designing recurrent networks systems that could perform long-ranged computations (such as outputting the first input it sees at the very end of an episode) by shaping its stable attractors.
For the general case, the intuition still holds. The situation is plotted in, Figures 3, 4, and 5.
Geometric model.
Continue using the above one-neuron network, fixing formula_63, and consider a loss function defined by formula_64. This produces a rather pathological loss landscape: as formula_65 approach formula_66 from above, the loss approaches zero, but as soon as formula_65 crosses formula_66, the attractor basin changes, and loss jumps to 0.50.
Consequently, attempting to train formula_65 by gradient descent would "hit a wall in the loss landscape", and cause exploding gradient. A slightly more complex situation is plotted in, Figures 6.
Solutions.
To overcome this problem, several methods were proposed.
Batch normalization.
Batch normalization is a standard method for solving both the exploding and the vanishing gradient problems.
Gradient clipping.
 recommends clipping the norm of formula_70 by formula_41:formula_72where formula_41 is the "threshold" hyperparameter, the maximum norm that the gradient is allowed to reach. Simply clipping each entry of formula_70 separately by formula_41 works as well in practice.
This does not solve the vanishing gradient problem.
Multi-level hierarchy.
One is Jürgen Schmidhuber's multi-level hierarchy of networks (1992) pre-trained one level at a time through unsupervised learning, fine-tuned through backpropagation. Here each level learns a compressed representation of the observations that is fed to the next level.
Related approach.
Similar ideas have been used in feed-forward neural networks for unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised backpropagation to classify labeled data. The deep belief network model by Hinton et al. (2006) involves learning the distribution of a high level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine to model each new layer of higher level features. Each new layer guarantees an increase on the lower-bound of the log likelihood of the data, thus improving the model, if trained properly. Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations. Hinton reports that his models are effective feature extractors over high-dimensional, structured data.
Long short-term memory.
Another technique particularly used for recurrent neural networks is the long short-term memory (LSTM) network of 1997 by Hochreiter &amp; Schmidhuber. In 2009, deep multidimensional LSTM networks demonstrated the power of deep learning with many nonlinear layers, by winning three ICDAR 2009 competitions in connected handwriting recognition, without any prior knowledge about the three different languages to be learned.
Faster hardware.
Hardware advances have meant that from 1991 to 2015, computer power (especially as delivered by GPUs) has increased around a million-fold, making standard backpropagation feasible for networks several layers deeper than when the vanishing gradient problem was recognized. Schmidhuber notes that this "is basically what is winning many of the image recognition competitions now", but that it "does not really overcome the problem in a fundamental way" since the original models tackling the vanishing gradient problem by Hinton and others were trained in a Xeon processor, not GPUs.
Residual networks.
One of the newest and most effective ways to resolve the vanishing gradient problem is with residual neural networks, or ResNets (not to be confused with recurrent neural networks). ResNets refer to neural networks where skip connections or residual connections are part of the network architecture. These skip connections allow gradient information to pass through the layers, by creating "highways" of information, where the output of a previous layer/activation is added to the output of a deeper layer. This allows information from the earlier parts of the network to be passed to the deeper parts of the network, helping maintain signal propagation even in deeper networks. Skip connections are a critical component of what allowed successful training of deeper neural networks.
ResNets yielded lower training error (and test error) than their shallower counterparts simply by reintroducing outputs from shallower layers in the network to compensate for the vanishing data. Note that ResNets are an ensemble of relatively shallow nets and do not resolve the vanishing gradient problem by preserving gradient flow throughout the entire depth of the network – rather, they avoid the problem simply by constructing ensembles of many short networks together. (Ensemble by Construction)
Other activation functions.
Rectifiers such as ReLU suffer less from the vanishing gradient problem, because they only saturate in one direction.
Weight initialization.
Weight initialization is another approach that has been proposed to reduce the vanishing gradient problem in deep networks.
Kumar suggested that the distribution of initial weights should vary according to activation function used and proposed to initialize the weights in networks with the logistic activation function using a Gaussian distribution with a zero mean and a standard deviation of codice_1, where codice_2 is the number of neurons in a layer.
Recently, Yilmaz and Poli performed a theoretical analysis on how gradients are affected by the mean of the initial weights in deep neural networks using the logistic activation function and found that gradients do not vanish if the "mean" of the initial weights is set according to the formula: codice_3. This simple strategy allows networks with 10 or 15 hidden layers to be trained very efficiently and effectively using the standard backpropagation.
Other.
Behnke relied only on the sign of the gradient (Rprop) when training his Neural Abstraction Pyramid to solve problems like image reconstruction and face localization.
Neural networks can also be optimized by using a universal search algorithm on the space of neural network's weights, e.g., random guess or more systematically genetic algorithm. This approach is not based on gradient and avoids the vanishing gradient problem.