that, ${\mbox{\boldsymbol{$W$}}}$ is learned in deconvolution but predefined in bilinear interpolation. Indeed bilinear interpolation has been shown to be a special case of deconvolution [@long2015fully]. Note that, in this case, the index function generates soft indices. The sense of index for the $\mathcal{PS}$ operator [@shi2016real] is also clear, because the rearrangement of the feature map is an indexing process. Considering $\mathcal{PS}$ a tensor $\mathcal{Z}$ of size $1\times1\times r^2$ to a matrix ${\mbox{\boldsymbol{$Z$}}}$ of size $r\times r$, the index function can be expressed by the one-hot encoding $$I^l_{ps}(x)=\mathbbm{1}(x= \mathcal{Z}_l)\,,l=1,...,r^2\,,$$ such that ${\mbox{\boldsymbol{$Z$}}}_{m,n}=\mathcal{Z}[I^l_{ps}(x)]$, where $m=1,...,r$, $n=1,...,r$, and $l=(r-1) \cdot m+n$. $\mathcal{Z}_l$ denotes the $l$-th element of $\mathcal{Z}$. Similar notation applies to ${\mbox{\boldsymbol{$Z$}}}_{m,n}$.

Since upsampling operators can be unified by the notion of the index function, it is plausible to ask whether one can learn an index function to dynamically capture local spatial patterns.

Learning to Index, to Pool, and to Upsample
===========================================

Before introducing the designs of IndexNet, we first present the general idea about how learned indices may be used in downsampling and upsampling with a new index-guided encoder-decoder framework. Our framework is a generalization of SegNet, as illustrated in Fig. \[fig:framework\]. For ease of exposition, let us assume the downsampling 