<p>So, let's start with: what is the aim of ResNets?</p>
<p>Given an input <code>X</code>, which is propagated through a certain ensemble of layers, let's call with <code>F(X)</code> the output of this ensemble. If we denote with <code>H(X)</code> the desired output (the ideal mapping, i.e. <code>F(X)!=H(X)</code>), a resnet learn <code>H(X) = F(X) + X</code>, that can be written as <code>F(X) = H(X)-X</code>, i.e the residual, from which the name residual network.</p>
<p>Thus, what is the gain of a resnet?</p>
<p>In a resnet, the mapping of a following layer performs at least as well as the previous one. Why? Because, at lest, it learns the mapping of an identity (<code>F(X)=X</code>).</p>
<p>This is a crucial aspect related to convolutional networks. Indeed, deeper nets should perform better than networks with lesser depth, but this does not always happen. From this rises the necessity to build a network that guarantees such behavior.</p>
<p>Is this true also for dense networks?
No, it is not. There is a known theorem (Universal Approximation Theorem) for dense nets, which states: any kind of network is equivalent to a two dense layers net with an adequate number of hidden units distributed between the two layers. For this reason, it is not necessary to increase the depth of a dense net, rather it is necessary to find the right number of hidden units.</p>
<p>If you want you can explore the original <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">paper</a> by He et al 2015.</p>
Does it make sense to build a residual network with only fully connected layers (instedad of convolutional layers)?
<p>Residual networks are always built with convolutional layers. I have never seen residual networks with only fully connected layers. Does it work to build a residual network with only fully connected layers?</p>