<p>The ReLU activation solves the problem of vanishing gradient that is due to sigmoid-like non-linearities (the gradient vanishes because of the flat regions of the sigmoid).</p>

<p>The other kind of "vanishing" gradient seems to be related to the depth of the network (<em>e.g.</em> see <a href="https://arxiv.org/abs/1803.01719" rel="noreferrer">this</a> for example). Basically, when backpropagating gradient from layer <code>N</code> to <code>N-k</code>, the gradient vanishes as a function of depth (in vanilla architectures). The idea of resnets is to help with gradient backpropagation (see for example <a href="https://arxiv.org/abs/1603.05027" rel="noreferrer">Identity mappings in deep residual networks</a>, where they present resnet v2 and argue that identity skip connections are better at this).</p>

<p>A very interesting and relatively recent paper that sheds light on the working on resnets is <a href="https://arxiv.org/abs/1605.06431" rel="noreferrer">resnets behaves as ensembles of relatively small networks</a>. The tl;dr of this paper could be (very roughly) summarized as this: residual networks behave as an ensemble: removing a single layer (i.e. a single residual branch, not its skip connection) doesn't really affect performance, but performance decreases in an smooth manner as a function of the number of layers that are removed, which is the way in which ensembles behave. Most of the gradient during training comes from short paths. They show that training only this short paths doesn't affect performance in a statistically significant way compared to when all paths are trained. This means that the effect of residual networks doesn't really come from depth as the effect of long paths is almost non-existant.</p>
Why is the main purpose of ResNet if the vanishing gradient problem is solved using RELU activation function?
<p>I read that ResNet solves the problem of vanishing gradient problem by using skip functions. But are they not already solved using RELU? Is there some other important thing I'm missing about ResNet or does Vanishing gradient problem occur even after using RELU? </p>