spatial and temporal dependencies of the traffic networks should be jointly modeled, so we integrate the proposed spatial and temporal transformer as a spatial-temporal transformer block, and residual connections are adopted among them as well. The structure of spatial-temporal block is illustrated in Fig. 3.

The input to the $l$-th spatial-temporal block is a 3D tensor $X^l\in {\mathbb R}^{M\times N\times d^l}$ with $M$ time steps of observations from $N$ nodes. Let ${\mathcal T}$ and $\mathcal S$ be the spatial and temporal transformers as aforementioned. Then the output $X^{l+1}\in {\mathbb R}^{M\times N\times d^{l+1}}$ can be formulated as $$X^{l+1}=X^l+{\mathcal S}(X^l,A)+{\mathcal T}(X^l+{\mathcal S}(X^l,A))$$ For simplicity, we denote $\mathcal S$ as spatial transformer and $\mathcal T$ as temporal transformer. Multiple spatial-temporal blocks can be stacked to improve the model capacity according tasks at hand.

### Prediction Layer

In the prediction layer, final time steps spatial-temporal features from the last spatial-temporal block are feed as input. And two stacked classical convolutional layers are then adopted to make multisteps predictions.

Formally, the input to the prediction layer is a 2D tensor $X^{\mathcal {ST}}\in {\mathbb R}^{N\times d^{{\mathcal {ST}}}}$, which is the last time step spatial-temporal features extracted by stacked spatial-temporal blocks. The final multi-steps prediction is $$Y=\text{Conv}(\text{Conv}(X^{\mathcal 