Title: $1\times1$ Convolution is All You Need for Image Super-Resolution. Abstract: In resource-constrained environments, such as mobile devices, lightweight and efficient architectures are crucial for the deployment of single image super-resolution (SISR) deep models. Due to the advantage of achieving a good trade-off between model capacity and efficiency, $3\times3$ convolutions are widely utilized in current convolutional neural networks (CNN). Compared to the normal $3\times3$ convolution, $1\times1$ convolution involves less computation burden but lacks the ability to represent and aggregate spatial information. Accordingly, a common sense in the literature is that  $1\times1$ solely cannot constitute a powerful SR network. In this paper, we revisit $1\times1$ in the lightweight scenario and demonstrate that the fully $1\times1$ convolutional network with strong learning ability can be achieved for SISR, thanks to the manual spatial-shift operation. We investigate the feature aggregation scheme in normal $3\times3$ convolution and analogously extend the $1\times1$ convolution with a parameter-free spatial-shift operation, simplified as the shift-conv layer. 
In the proposed SISR method, we replace all normal $3\times3$ convolutions with shift-conv layers and present the $\mathbf{S}$hift-$\mathbf{C}$onv-based $\mathbf{N}$etwork (SCNet). Extensive experiments demonstrate that SCNets with all $1\times1$ convolutions obtain even better results than SR models with normal $3\times3$ convolutions that have a larger model size.