(Eq.6).

Then, the whole spatial transformer can be summarized as $$Y^{\mathcal S}={\mathcal S}(X, A)$$

In summary, the proposed spatial transformer has the following properties. 1) Dynamical hidden spatial dependencies are calculated in the learned high-dimensional latent subspaces, and prior graph structure information can be incorporated via residual structures. 2)The learned hidden spatial dependencies are not restricted to local nodes but can be globally, and multiple spatial dependencies can be explored in different latent relationship spaces. 3)It is fully feed-forward, thus computations can be performed in parallel in a fast and scalable fashion, in other words, we can easily scale our model to large-scale traffic forecasting with proper distribution computation scheme.

Temporal Transformers
---------------------

![image](dependencies.pdf){width="2\columnwidth" height="1\columnwidth"}

We also develop a temporal transformer to efficiently and effectively capture long-range temporal dependencies [@NIPS2017_7181] over time. Compared with RNNs and the variants, it can not only capture long-range dependencies but also allow parallel calculations so that it can be easily scaled to long sequences. The structures are illustrated in the bottom right of Fig.3.

### Self-attention Layer for Long-range Modeling

Similar to temporal position adopted in spatial transformer, we first concatenate the learned spatial features in each time step with temporal embedding which is 