<p>You can't do it solely using <code>torch.nn.Sequential</code> as it requires operations to go, as the name suggests, sequentially, while yours are parallel.</p>
<p>You could, in principle, construct your own <code>block</code> really easily like this:</p>
<pre><code>import torch

class ResNet(torch.nn.Module):
    def __init__(self, module):
        super().__init__()
        self.module = module

    def forward(self, inputs):
        return self.module(inputs) + inputs
</code></pre>
<p>Which one can use something like this:</p>
<pre><code>model = torch.nn.Sequential(
    torch.nn.Conv2d(3, 32, kernel_size=7),
    # 32 filters in and out, no max pooling so the shapes can be added
    ResNet(
        torch.nn.Sequential(
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
        )
    ),
    # Another ResNet block, you could make more of them
    # Downsampling using maxpool and others could be done in between etc. etc.
    ResNet(
        torch.nn.Sequential(
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
        )
    ),
    # Pool all the 32 filters to 1, you may need to use `torch.squeeze after this layer`
    torch.nn.AdaptiveAvgPool2d(1),
    # 32 10 classes
    torch.nn.Linear(32, 10),
)
</code></pre>
<p>Fact being usually overlooked (without real consequences when it comes to shallowe networks) is that skip connection should be left <strong>without</strong> any nonlinearities like <code>ReLU</code> or convolutional layers and that's what you can see above (source: <a href="https://arxiv.org/pdf/1603.05027.pdf" rel="noreferrer">Identity Mappings in Deep Residual Networks</a>).</p>
How to implement my own ResNet with torch.nn.Sequential in Pytorch?
<p>I want to implement a ResNet network (or rather, <strong>residual blocks</strong>) but I really want it to be in the sequential network form.</p>
<p>What I mean by sequential network form is the following:</p>
<pre><code>## mdl5, from cifar10 tutorial
mdl5 = nn.Sequential(OrderedDict([
    ('pool1', nn.MaxPool2d(2, 2)),
    ('relu1', nn.ReLU()),
    ('conv1', nn.Conv2d(3, 6, 5)),
    ('pool1', nn.MaxPool2d(2, 2)),
    ('relu2', nn.ReLU()),
    ('conv2', nn.Conv2d(6, 16, 5)),
    ('relu2', nn.ReLU()),
    ('Flatten', Flatten()),
    ('fc1', nn.Linear(1024, 120)), # figure out equation properly
    ('relu4', nn.ReLU()),
    ('fc2', nn.Linear(120, 84)),
    ('relu5', nn.ReLU()),
    ('fc3', nn.Linear(84, 10))
]))
</code></pre>
<p>but of course with the NN lego blocks being “ResNet”.</p>
<p>I know the equation is something like:</p>
<p><a href="https://i.stack.imgur.com/gn2va.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gn2va.png" alt="enter image description here" /></a></p>
<p>but I am not sure how to do it in Pytorch AND Sequential. Sequential is key for me!</p>
<hr />
<h1>Bounty:</h1>
<p>I'd like to see an example with a fully connected net and where the BN layers would have to go (and the drop out layers would go too). Ideally on a toy example/data if possible.</p>
<hr />
<p>Cross-posted:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-have-residual-network-using-only-sequential-blocks/51541" rel="nofollow noreferrer">https://discuss.pytorch.org/t/how-to-have-residual-network-using-only-sequential-blocks/51541</a></li>
<li><a href="https://www.quora.com/unanswered/How-does-one-implement-my-own-ResNet-with-torch-nn-Sequential-in-Pytorch" rel="nofollow noreferrer">https://www.quora.com/unanswered/How-does-one-implement-my-own-ResNet-with-torch-nn-Sequential-in-Pytorch</a></li>
<li><a href="https://www.reddit.com/r/pytorch/comments/uyyr28/how_to_implement_my_own_resnet_with/" rel="nofollow noreferrer">https://www.reddit.com/r/pytorch/comments/uyyr28/how_to_implement_my_own_resnet_with/</a></li>
</ul>