Fig. \[fig:indexnet\_taxonomy\] shows the tree diagram of these index networks. The difference between modelwise IndexNet and stagewise IndexNet is also shown in Fig. \[fig:modelwise\_stagewise\]. Notice that, HINs and M2O DINs are both stagewise.

With the taxonomy, we investigate five families of IndexNet. Each family can be designed to have either linear mappings or nonlinear mappings, as we discuss next.

![Modelwise IndexNet vs.  stagewise IndexNet.[]{data-label="fig:modelwise_stagewise"}](modelwise_stagewise.pdf){width="3.4in"}

Holistic Index Networks {#subsec:holistic_networks}
-----------------------

Recall that HINs learn an index function $I(\mathcal{X}):\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{H\times W\times1}$. A naive design choice is to assume a linear mapping between the feature map and the index map.

*Linear HINs.* An example is shown in Fig. \[fig:holistic\_networks\](a). The network is implemented in a fully convolutional network. It first applies stride-$2$ $2\times2$ convolution (assuming that the downsampling rate is $2$) to the feature map of size $H\times W\times C$, generating a concatenated index map of size $H/2\times W/2\times 4$. Each slice of the index map ($H/2\times W/2\times 1$) is designed to correspond to the indices of a certain position of all local regions, e.g., the top-left corner of all $2\times 2$ regions. The network finally applies a $\mathcal{PS}$-like shuffling operator to rearrange the index map to the size of $H\times W\times 