e.g., a $\max$ function, or more generally, a parameterized function such as neural network. In this work, we use a fully convolutional network to be the index block. More details are presented in Sections \[subsec:holistic\_networks\] and \[subsec:depthwise\_networks\]. Note that the index maps sent to the encoder and decoder are normalized differently. The decoder index map only goes through a *sigmoid* function such that for any predicted index $i\in(0,1)$. As for the encoder index map, indices of each local region $L$ are further normalized by a *softmax* function such that $\sum_{i\in{L}}i=1$. The second normalization guarantees the magnitude consistency of the feature map after downsampling.

*Indexed Pooling* ($\mathcal{IP}$) performs downsampling using generated indices. Given a local region $E\in\mathbb{R}^{k\times k}$, $\mathcal{IP}$ calculates a weighted sum of activations and corresponding indices over $E$ as $\mathcal{IP}(E)=\sum_{x\in E}I(x)x$, where $I(x)$ is the index of $x$. It is easy to see that max pooling and average pooling are special cases of $\mathcal{IP}$. In practice, this operator can be easily implemented with an element-wise multiplication between the feature map and the index map, an average pooling layer, and a multiplication of a constant used to compensate the effect of averaging, as instantiated in Fig. \[fig:framework\]. The current implementation 