correspond to different assumptions. We compare the behavior of IndexNet with existing upsampling operators and demonstrate its superiority in delivering spatial information. We show that IndexNet can be incorporated into many CNNs to benefit a number of visual tasks, for instance: i) *image matting*: our MobileNetv2-based [@sandler2018mobilenetv2] model with IndexNet exhibits at least $16.1\%$ improvement against the VGG-16-based DeepMatting baseline [@xu2017deep] on the Composition-1k matting dataset; by visualizing learned indices, the indices automatically learn to capture the boundaries and textural patterns; ii) *image denoising*: a modified DnCNN model with IndexNet can achieve performance comparable to the baseline DnCNN [@zhang2017beyond] that has no downsampling stage on the BSD68 and Set12 datasets [@roth2009fields], thus reducing the computational cost and memory consumption significantly; iii) *semantic segmentation*: consistently improved performance is observed when SegNet [@badrinarayanan2017segnet] is equipped with IndexNet on the SUN RGB-D dataset [@song2015sun]; and iv) *monocular depth estimation*: IndexNet also improves the performance of a recent light-weight FastDepth model on the NYUDv2 dataset [@silberman2012indoor], with negligible extra computation cost.

We make the following main contributions:

-   We present a unified perspective of existing upsampling operators with the notion of the index function;

-   We introduce Index Networks—a novel family of networks that 