we also include the choice of the activation function in the operation level search space and rely on the search algorithm to choose the right one for each attention operation. We consider the following four choices for the activation function: (1) no activation function, (2) ReLU, (3) sigmoid, and (4) softmax.

Cell Level Search Space
-----------------------

We define an attention cell as a cell composed of $K$ attention operations. Let $f_0$ denote the input feature map to the entire attention cell and $(T, H, W, C)$ be the shape of $f_0$. $f_0$ is usually the output of a stack of convolutional layers. An attention cell takes $f_0$ as input and outputs a feature map of the same shape.

The connectivity between convolutional layers is essential to the performance of CNNs, no matter if the network is manually designed, e.g., ResNet [@he2016deep] and Inception [@szegedy2015going], or automatically discovered [@zoph2016neural; @zoph2018learning; @xie2019exploring]. Similarly, to build an attention cell, another critical design choice is how the $K$ attention operations are connected inside the cell, apart from the design of these attention operations.

As shown in Fig. \[fig-search-space\], in an attention cell, the first attention operation always takes $f_0$ as input and outputs feature 