the feature map of size $H\times W\times Cr^2$ to $rH\times rW\times C$. It is also used in some segmentation models [@yang2019deeperlab].

Our work is primarily inspired by the unpooling operator [@badrinarayanan2017segnet]. We remark that, it is important to extract spatial information before its loss during downsampling, and more importantly, to use stored information during upsampling. Unpooling shows a simple and effective use case, while we believe that there is much room to improve. Here we show that unpooling is a special form of index function, and we can learn an index function beyond unpooling.

We notice that concurrent work of [@jiaqi2019carafe] also pursues the idea of data-dependent upsampling and proposes an universal upsampling operator termed CARAFE. Although the idea is similar, IndexNet is different from CARAFE in several aspects. First, CARAFE does not associate upsampling with the notion of the index function. Second, the kernels used in CARAFE are generated conditioned on decoder features, while IndexNet builds upon encoder features, so the generated indices can also be used to guide downsampling. Third, CARAFE can be viewed as one of our investigated index networks—holistic index networks, but with different upsampling kernels and normalization strategies. We further compare IndexNet with CARAFE in 