long sequences. Alternatively, [@yu2018spatio] [@guo2019attention] adopted Convolution-based sequence learning models [@gehring2017convolutional], however, the limited size of receptive fields requires multiple hidden layers to cover a sufficiently large context. [@wu2019graph] adopted WaveNet with dilation convolution to increase the receptive field to cover the whole sequence with smaller layers. However, the number of layers still increase linearly with input sequence length, thus limiting the model scalability to explore long input sequence. Furthermore, deep layers increases the length of the path between two components, and degrades its efficiency in capturing long-range dependencies [@hochreiter2001gradient] [@NIPS2017_7181]. The model needs to be redesigned if the input sequence length are changed, thus making it expensive to search for optimal input sequence length. In contrast, transformers [@NIPS2017_7181] are a newly proposed efficient sequence learning model that relies on a self-attention mechanism that is highly parallelizable. It enables to effectively and efficiently capture long-range dependencies over time with single layers and can be easily adapted for different input sequence length.

The Formulation
===============

A traffic network can naturally be represented as a graph ${\mathcal G}=({\mathcal V, \mathcal E }, A)$, where ${\mathcal V}$ is the set of nodes representing the sensors with $|{\mathcal V}|=N$, ${\mathcal E}$ is the set 