map $f_1$. The $k^{th} (2 \leq k \leq K)$ attention operation chooses its input from $\{f_0, f_1, \ldots, f_{k-1}\}$ and gives feature map $f_k$ based on the selected input. We allow the $k^{th}$ operation to choose multiple feature maps from $\{f_0, f_1, \ldots, f_{k-1}\}$ and compute a weighted sum of selected feature maps as its input, where the weights are learnable parameters. This process is repeated for all $k$ and allows us to explore all possible connectivities between the $K$ attention operations in the cell.

We combine $\{f_1, f_2, \ldots, f_K\}$ to obtain the output feature map of the entire attention cell. For all attention operations inside the cell, we set their output shape to be $(T, H, W, C_\text{op})$, i.e., $f_k$ has the shape of $(T, H, W, C_\text{op})$ for all $k (1 \leq k \leq K)$. $C_{\text{op}}$ is usually smaller than $C$ to limit the computation in an attention cell with multiple attention operations. We concatenate $\{f_1, f_2, \ldots, f_K\}$ along the channel dimension and then employ a $1\times 1 \times 1$ convolution to transform the concatenated feature map back to the same shape as the input $f_0$. We denote the feature map after transformation as $f_\text{comb}$. Similar 