Title: Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck. Abstract: In this paper, we present an in-depth investigation of the convolutional autoencoder (CAE) bottleneck.
Autoencoders (AE), and especially their convolutional variants, play a vital role in the current deep learning toolbox.
Researchers and practitioners employ CAEs for various tasks, ranging from outlier detection and compression to transfer and representation learning.
Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the CAE's emergent properties.
We demonstrate that increased bottleneck area (i.e., height $\times$ width) drastically improves generalization in terms of reconstruction error while also speeding up training.
The number of channels in the bottleneck, on the other hand, is of secondary importance.
Furthermore, we show empirically that CAEs do not learn to copy their input, even when all layers have the same number of neurons as there are pixels in the input (i.e. there is no bottleneck).
Besides raising important questions for further research, our findings are directly applicable to two of the most common use-cases for CAEs:
In image compression, it is advantageous to increase the feature map size in the bottleneck as this greatly improves reconstruction quality.
For reconstruction-based outlier detection, we recommend decreasing the feature map size so that out-of-distribution samples will yield a higher reconstruction error.