We reshape $f_\text{in}$ into a 2D feature map $f'_\text{in}$ before computing the attention weights. The first dimension of $f'_\text{in}$ is the attention dimension and the second dimension contains the remaining dimensions. For example, $f'_\text{in}$ has the shape of $(T, HWC_{\text{in}})$ when temporal is the attention dimension and has the shape of $(THW, C_{\text{in}})$ when spatiotemporal is the attention dimension. We denote this procedure as a function $\texttt{ReshapeTo2D}$, i.e., $f'_\text{in} = \texttt{ReshapeTo2D}(f_\text{in})$.

Spatial attention requires extra handling. As video content changes over time, when applying attention to the spatial dimension, each frame $f_\text{in}^t$ should have its own spatial attention weights, where $f_\text{in}^t$ is the $t^{th}$ frame in $f_\text{in}$ and has the shape of $(H, W, C_\text{in})$. Therefore, when spatial is the attention dimension, instead of reshaping the entire 4D feature map $f_\text{in}$, we reshape $f_\text{in}^t$ into a 2D feature map $f_\text{in}^{\prime t}$ of shape $(HW, C_\text{in})$ for every $t$, i.e., $f_\text{in}^{\prime t}=\texttt{ReshapeTo2D}(f_\text{in}^t) (1 \leq t \leq T)$.

#### Map-based attention.

Assuming temporal is the attention dimension, map-based attention generates $T$ attention weights to scale the feature map of each temporal frame. The attention weights are computed as follows: $$\label{eq-mapatt}
W_\text{map} = \texttt{Diag}(\phi(G_2(\texttt{AvgPool}(G_1(f'_\text{in}))))).$$ $G_1$ is a 1D convolutional layer with kernel size 