<h2><a href="https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks">Dying ReLU</a></h2>
<ul>
<li>I think the main reason for <strong>underfitting</strong> in your case is <a href="https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks"><strong>Dying Relu</strong></a> problem. Your network is simple Autoencoder with <a href="https://theaisummer.com/skip-connections/" rel="nofollow noreferrer"><strong>no skip/residual</strong></a> connections. So <strong>Code</strong> in the bottleneck should encode enough information about <strong>bias</strong> in the data to make <strong>Decoder</strong> to learn.</li>
<li>So if <strong>ReLU</strong> activation function is used <strong>Negative Biased</strong> data information can be <strong>lost</strong> due to Dying ReLU problem. The solution is to to use better activation functions like <strong>LeakyReLU</strong>, <strong>ELU</strong>, <a href="https://arxiv.org/abs/1908.08681" rel="nofollow noreferrer"><strong>MISH</strong></a>, etc.</li>
</ul>
<h2>Linear vs Conv.</h2>
<p>In your case, you are <strong>overfitting</strong> on a single batch. As <strong>Linear</strong> layers will have more <strong>parameters</strong> than that of <strong>Convolution</strong> layers maybe they are <strong>Memorising</strong> given small data easily.</p>
<h2>Batch Size</h2>
<p>As you are <strong>overfitting</strong> on a single batch, a <strong>small-batch</strong> of data will make it very easy to <strong>memorise</strong> on the other hand for <strong>large batch</strong> with single <strong>Update</strong> of network per batch(during overfitting) make network to learn <strong>Generalized</strong> abstract features. (This works better if more batches are there with a lot of variety of data)</p>
<p>I tried to reproduce your problem using simple <strong>Gaussian</strong> data. Just by using <strong>LeakyReLU</strong> in place of <strong>ReLU</strong> with proper learning rate solved the problem.
Same architecture given by you is used.</p>
<p>Hyper parameters:</p>
<p>batch_size = 16</p>
<p>epochs = 100</p>
<p>lr = 1e-3</p>
<p>optimizer = Adam</p>
<p>loss(after training with <strong>ReLU</strong>) = 0.27265918254852295</p>
<p>loss(after training with <strong>LeakyReLU</strong>) = 0.0004763789474964142</p>
<h2>With Relu</h2>
<p><a href="https://i.stack.imgur.com/muhl5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/muhl5.png" alt="with relu" /></a></p>
<h2>With Leaky Relu</h2>
<p><a href="https://i.stack.imgur.com/Ve7ww.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Ve7ww.png" alt="with Leaky relu" /></a></p>
Underfitting a single batch: Can't cause autoencoder to overfit multi-sample batches of 1d data. How to debug?
<h2><strong>TL;DR</strong></h2>
<p>I am unable to overfit batches with multiple samples using autoencoder.</p>
<p>Fully connected decoder seems to handle more samples per batch than conv decoder, but then also fails when number of samples increases.
<strong>Why is this happening, and how to debug this?</strong></p>
<hr />
<h2>In depth</h2>
<p>I am trying to use an auto encoder on 1d data points of size <code>(n, 1, 1024)</code>, where <code>n</code> is the number of samples in the batch.</p>
<p>I am trying to overfit to that single batch.</p>
<p>Using a convolutional decoder, I am only able to fit a single sample (<code>n=1</code>), and when <code>n&gt;1</code> I am unable to drop the loss (MSE) below 0.2.</p>
<p><strong>In blue: expected output (=input), in orange: reconstruction.</strong></p>
<p>Single sample, single batch:<br />
<a href="https://i.stack.imgur.com/IywNA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IywNA.png" alt="Conv1sample" /></a></p>
<p>Multiple samples, single batch, loss won't go down:
<a href="https://i.stack.imgur.com/OscSR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OscSR.png" alt="Conv4samples" /></a></p>
<p>Using more than one sample, we can see the net learns the general shape of the input (=output) signal, but greatly misses the bias.</p>
<hr />
<p>Using a fully connected decoder does manage to reconstruct batches of multiple samples:</p>
<p><a href="https://i.stack.imgur.com/reik6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/reik6.png" alt="Fc4samples" /></a></p>
<hr />
<h2>Relevant code:</h2>
<pre><code>class Conv1DBlock(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size):
        super().__init__()
        self._in_channels = in_channels
        self._out_channels = out_channels
        self._kernel_size = kernel_size

        self._block = nn.Sequential(
                nn.Conv1d(
                        in_channels=self._in_channels,
                        out_channels=self._out_channels,
                        kernel_size=self._kernel_size,
                        stride=1,
                        padding=(self._kernel_size - 1) // 2,
                ),
                # nn.BatchNorm1d(num_features=out_channels),
                nn.ReLU(True),
                nn.MaxPool1d(kernel_size=2, stride=2),
        )

    def forward(self, x):
        for layer in self._block:
            x = layer(x)
        return x


class Upsample1DBlock(nn.Module):
    def __init__(self, in_channels, out_channels, factor):
        super().__init__()
        self._in_channels = in_channels
        self._out_channels = out_channels
        self._factor = factor

        self._block = nn.Sequential(
                nn.Conv1d(
                        in_channels=self._in_channels,
                        out_channels=self._out_channels,
                        kernel_size=3,
                        stride=1,
                        padding=1
                ),  # 'same'
                nn.ReLU(True),
                nn.Upsample(scale_factor=self._factor, mode='linear', align_corners=True),
        )

    def forward(self, x):
        x_tag = x
        for layer in self._block:
            x_tag = layer(x_tag)
        # interpolated = F.interpolate(x, scale_factor=0.5, mode='linear') # resnet idea
        return x_tag
</code></pre>
<p>encoder:</p>
<pre><code>self._encoder = nn.Sequential(
            # n, 1024
            nn.Unflatten(dim=1, unflattened_size=(1, 1024)),
            # n, 1, 1024
            Conv1DBlock(in_channels=1, out_channels=8, kernel_size=15),
            # n, 8, 512
            Conv1DBlock(in_channels=8, out_channels=16, kernel_size=11),
            # n, 16, 256
            Conv1DBlock(in_channels=16, out_channels=32, kernel_size=7),
            # n, 32, 128
            Conv1DBlock(in_channels=32, out_channels=64, kernel_size=5),
            # n, 64, 64
            Conv1DBlock(in_channels=64, out_channels=128, kernel_size=3),
            # n, 128, 32
            nn.Conv1d(in_channels=128, out_channels=128, kernel_size=32, stride=1, padding=0),  # FC
            # n, 128, 1
            nn.Flatten(start_dim=1, end_dim=-1),
            # n, 128
        )
</code></pre>
<p>conv decoder:</p>
<pre><code>self._decoder = nn.Sequential(
    nn.Unflatten(dim=1, unflattened_size=(128, 1)),  # 1
    Upsample1DBlock(in_channels=128, out_channels=64, factor=4),  # 4
    Upsample1DBlock(in_channels=64, out_channels=32, factor=4),  # 16
    Upsample1DBlock(in_channels=32, out_channels=16, factor=4),  # 64
    Upsample1DBlock(in_channels=16, out_channels=8, factor=4),  # 256
    Upsample1DBlock(in_channels=8, out_channels=1, factor=4),  # 1024
    nn.ReLU(True),
    nn.Conv1d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=1),
    nn.ReLU(True),
    nn.Flatten(start_dim=1, end_dim=-1),
    nn.Linear(1024, 1024)
)
</code></pre>
<p>FC decoder:</p>
<pre><code>self._decoder = nn.Sequential(
    nn.Linear(128, 256),
    nn.ReLU(True),
    nn.Linear(256, 512),
    nn.ReLU(True),
    nn.Linear(512, 1024),
    nn.ReLU(True),
    nn.Flatten(start_dim=1, end_dim=-1),
    nn.Linear(1024, 1024)
)
</code></pre>
<hr />
<p>Another observation is that when the batch size increases more, to say, 16, the FC decoder also starts to fail.</p>
<p>In the image, 4 samples of a 16 sample batch I am trying to overfit</p>
<p><a href="https://i.stack.imgur.com/jE1Pj.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jE1Pj.png" alt="fc16Samples" /></a></p>
<hr />
<p>What could be wrong with the conv decoder?</p>
<p>How to debug this or make the conv decoder work?</p>