Weakness - I am not convinced that transformer free of locality-bias is indeed the best option. In fact, due to limited speed of information propagation, the neighborhood agents should naturally have more impacts on each other, compared to far away nodes. I hope the authors to explain more why transformer’s no-locality won’t make a concern here. - Due to the above, I feel graph networks seem to capture this better than the too-free transformer, and their lack of global context/ the “over-squashing” might be mitigated by adding non-local blocks (e.g., check “Non-Local Graph Neural Networks” or several other works proposing “global attention” for GNNs). - The authors also claimed “traditional GNNs” cannot handle direction-feature coupling: that is not true. See a latest work “MagNet: A Neural Network for Directed Graphs” and I am sure there were more prior arts. Authors are asked to consider whether those directional GNNs can possibly suit their task well too. - Transform is introduced as a centralized agent. Its computational overhead can become formidable when the network gets larger. Authors shall discuss how they prepare to address the scalability bottleneck.
- I am not convinced that transformer free of locality-bias is indeed the best option. In fact, due to limited speed of information propagation, the neighborhood agents should naturally have more impacts on each other, compared to far away nodes. I hope the authors to explain more why transformer’s no-locality won’t make a concern here.