$\phi_k$. The only difference among these attended feature maps $\{f_{i,j}^{\text{out}, \phi_k}\}$ is the activation function $\phi$ used in Eq. \[eq-mapatt\] or Eq. \[eq-dotproductatt\]. The layers $G_1$, $G_2$ and $G_3$ are shared by different activation functions within one node.

The supergraph has a sink node, receiving the output feature maps of all the nodes. The sink node is defined as follows: $$f_{\text{sink}}^{\text{out}} = \sum_{1\leq i \leq m, 1\leq j \leq n} w_{i, j}^{\text{sink}} \cdot G_{i,j}(f_{i, j}^{\text{out}}),$$ where $f_{\text{sink}}^{\text{out}}$ is the output of the sink node, $f_{i, j}^{\text{out}}$ is the output of the $j^{th}$ node at $i^{th}$ level, $G_{i,j}$ is a $1 \times 1 \times 1$ convolutional layer changing the number of channels in $f_{i, j}^{\text{out}}$ to $C$, and $w_{i, j}^{\text{sink}}$ is the weighting factor to be learned. We enforce $f_{\text{sink}}^{\text{out}}$ to have the same shape as the input to the supergraph, so that the supergraph can be inserted into any position of the backbone network. Same as attention cells, a residual connection is added between the input and output of the supergraph.

### Attention Cell Design Learning

Both the network weights, e.g., weights of convolutional layers in the network, and the connection weights in the supergraph ($\{w^{\text{level}}, w^{\text{sink}}, w^{\text{activation}} \}$) are 