as 1, which reduces the dimension of the feature response of each temporal frame from $HWC_{\text{in}}$ to $C'$ and gives a feature map of shape $(T, C')$. $\texttt{AvgPool}$ denotes an average pooling operation applied to each temporal dimension and outputs a $T$-dim vector. The multilayer perceptron $G_2$ and the activation function $\phi$ (e.g., the sigmoid function) further transform the $T$-dim vector to $T$ attention weights. More details about the activation function are discussed later. $\texttt{Diag}$ rearranges the $T$ attention weights into a $T \times T$ matrix, where the $T$ attention weights are placed on the diagonal of the matrix. The obtained attention weight matrix $W$ is a diagonal matrix.

Similarly, when spatiotemporal is the attention dimension, map-based attention gives a $THW \times THW$ diagonal matrix containing the attention weights. When spatial is the attention dimension, we generate one $HW \times HW$ diagonal matrix for every $f_{\text{in}}^{\prime t}$ ($1\leq t \leq T$) separately, using the above described procedure. Note that while different frames have separate spatial attention weights, $G_1$ and $G_2$ are shared among different frames when computing attention weights.

#### Dot-product attention.

When applying dot-product attention to the temporal dimension, a $T\times T$ attention weight matrix is generated as follows: 