### Attention Cell Design Derivation

We derive the attention cell design from the learned continuous connection weights. We first choose the top $\alpha$ nodes with the highest weights in $w^{\text{sink}}$ and add them to the set $S$. Then for each node in $S$, we add its top $\beta$ predecessors in its previous level to $S$, based on the corresponding connection weights in $w^{\text{level}}$. This process is conducted recursively for every node in $S$ until we reach the first level. $\alpha$ and $\beta$ are two hyper-parameters.

Recall that each node is an attention operation of a pre-defined type and attention dimension. So, $S$ contains a set of selected attention operations. The construction process of $S$ also determines how these attention operations are connected. For all the selected attention operations, we decide its activation function based on the corresponding weighting factors in $w^{\text{activation}}$.

Experiments
===========

Experimental Setup
------------------

**Datasets.** We conduct experiments on two benchmark datasets: Kinetics-600 [@carreira2018short] and Moments in Time (MiT) [@monfort2019moments]. Top-1 and top-5 classification accuracy are used as the evaluation metric for both datasets.

**Backbones.** We conduct the attention cell search on two backbones: I3D [@carreira2017quo] and S3D [@xie2018rethinking]. Both I3D and S3D are constructed based on the 