further calculated by dot-product as $$S^{\mathcal S}=\text{softmax}(\frac{Q^{\mathcal S}(K^{\mathcal S})^T}{\sqrt{d^{\mathcal S}_k}})$$ where $S^{\mathcal S}$ is the learned dynamical dependencies matrix among nodes. Then, new node features $M^s\in{\mathbb R^{N\times d^{\mathcal S}_v}}$ are further updated with $$M^s=S^{\mathcal S}V_{\mathcal S}$$ Note that we adopt dot product to calculate dependencies between nodes as it is much fast and space-efficient in practice. The softmax is then applied to normalize the dependencies. Scaling by $\sqrt{d^{\mathcal S}_k}$ is to avoid the softmax from reaching the saturation when gradients are extremely small. Multiple dependencies can be learned with multi-heads attention mechanism by learning multiple subspace pairs, thereby revealing different hidden spatial dependencies from various latent subspaces.

To further improve the prediction ability of learned node features, a shared three-layer feed-forward neural network with nonlinear activation is applied on each node to explore the interactions among features channels and update the node features as $Y^s\in {\mathbb R}^{N \times d^s_o}$. $$U^{\mathcal S}=\text{ReLu}(\text{ReLu}(M^{\mathcal S}W^{\mathcal S}_0)W^{\mathcal S}_1)W^{\mathcal S}_2$$ where $W^{\mathcal S}_0, W^{\mathcal S}_1, W^{\mathcal S}_2$ are the weight matrices, and $U^{\mathcal S}=X^{\mathcal S}+M^{\mathcal S}$ is the residual connection for stable training.

To model more complicated spatial dependencies, we can stack several dynamical graph convolution layers for deep model to improve the model capacity.

