<p>I would refer you to <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf" rel="nofollow noreferrer">the original paper</a> by Kaiming He at al.</p>

<p>In sections 3.1-3.2, they define "identity" shortcuts as <code>y = F(x, W) + x</code>, where <code>W</code> are the trainable parameters, <strong>for any residual mapping</strong> <code>F</code> to be learned. It is important that the residual mapping contains a <em>non-linearity</em>, otherwise the whole construction is one sophisticated linear layer. But the number of linearities is not limited. </p>

<p>For example, <a href="https://arxiv.org/pdf/1611.05431.pdf" rel="nofollow noreferrer">ResNeXt network</a> creates identity shortcuts around a stack of only convolutional layers (see the figure below). So there aren't <em>any</em> dense layers in the residual block.</p>

<p><a href="https://i.stack.imgur.com/A5zqg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/A5zqg.png" alt="res-next-blocks"></a></p>

<p>The general answer is, thus: yes, it would work. However, in a particular neural network, reducing two dense layers to one may be a bad idea, because anyway the residual block must be flexible enough to learn the residual function. So remember to validate any design you come up with.</p>
Can Residual Nets skip one linearity instead of two?
<p>Standard in ResNets is to skip 2 linearities.
Would skipping only one work as well?</p>