$f_\text{in}$ from $C_\text{in}$ to $C_\text{out}$. If temporal is the attention dimension, $W$ has the shape of $(T, T)$ and $\texttt{ReshapeTo2D}(G_3(f_\text{in}))$ has the shape $(T, HWC_\text{out})$. $\texttt{ReshapeTo2D}^{-1}$ is the inverse function of $\texttt{ReshapeTo2D}$, reshaping the attended feature map back to the shape of $(T, H, W, C_\text{out})$.

For spatial attention, the attention weights are applied to each frame independently, i.e., $f^t_\text{out}  = \texttt{ReshapeTo2D}^{-1} (  W^t \texttt{ReshapeTo2D}(G_3(f^t_\text{in})) )$, where $W^t$ is the spatial attention weights for frame $t$ and $f^t_\text{out}$ has the shape of $(H, W, C_\text{out})$. We stack $\{f^t_\text{out}\mid 1\leq t \leq T\}$ along the temporal dimension to form the attended feature map $f_\text{out}$ of shape $(T, H, W, C_\text{out})$. Similar to $G_1$ and $G_2$ used for computing attention weights, $G_3$ is also shared among different frames.

Note that by design $G_3$ only changes number of channels, i.e., transforms the features at each spatiotemporal position. The spatiotemporal structure of the input $f_\text{in}$ is preserved. This ensures that after the application of attention weights, $f_\text{out}$ still follows the original spatiotemporal structure of the input $f_\text{in}$.

#### Activation function.

We empirically find that the activation function $\phi$ (see Eq. \[eq-mapatt\] and Eq. \[eq-dotproductatt\]) used in the attention operation can influence the performance. So, 