Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD).

The review comment to be evaluated is:

Strength
- The method is motivated by the failure of current architectures over a synthetic task. The proposed method brings significant improvement to it.
-  The method demonstrates empirical gains over some datasets, mostly on ones that requires capturing long-range interactions

Weaknesses
- Some important parts of the paper are confusing, mostly due to the ordering of the sections and the phrases:
    - It is not clear to me what "learn to combine information propagation over multiple different propagation steps in an
end-to-end manner" means. What are the "multiple different propagation steps"?
    - The use of the phrase "virtual edge" is somewhat confusing. Based on the later formulas, it seems these virtual edges are essentially generated features between all pair of nodes (based on 4.1), and replace the original edges/edge features. But the phrase itself and Figure 1 makes it seem like the virtual edges are added to the original set of edges.
- Some of the statements seem to be over-claimed:
    - "However, given the arbitrary structure of graphs, incorporating the input into the Transformer remains a challenging aspect in designing GTs, and so far, there has been no universal solution. We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)..." It should be clarified that a lot of the current graph transformers designs are expressive enough incorporate arbitrary structure of graphs (e.g. Graphormer, GRAPHGPS, etc) and showed strong results. The authors should narrow down the claim to improving the modeling of long-range interaction.
    - "this work is the first Graph Transformer to [...] learn to combine information propagation over multiple different propagation steps in an end-to-end manner." It is not particularly clear to me what this statement means, but seems this is referring to feeding stacked random walk matrices through feed-forward neural networks in 4.1.2. I believe this has already been mentioned in GRAPHGPS and GraphiT (i.e. Relative PE using random walks).
- The first step of learning a dense adjacency matrix could be very expensive for large graphs.
- The empirical gain on ZINC and most of the OGB datasets seem quite small compared to the variance.

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

We now describe our approach for taking a graph \(G=(X,A)\) with a node embeddings matrix \(X\) and edges \(A\) and processing it using the Transformer, which usually takes only a single matrix \(X\) as input. Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

We stack different powers of \(A\) into a 3 dimensional tensor \(E\in R^{n\times n\times k}\) \[E=[I|A|A^{2}|..|A^{k}] \tag{1}\] where \(k\) is the number of stacks and \(|\) denotes stacking matrices along the 3ed dimension and \(I\) is the identity matrix.

Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network. Each layer consists of 1 hidden layer with batch norm, ReLU activation, and a residual connection. \[Edge\text{-}FFN(E_{ij})=ReLU(BN(E_{ij}W1))W2^{1} \tag{2}\] \[E_{ij}=Edge\text{-}FFN(E_{ij})+E_{ij} \tag{3}\]

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself. Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\). \[\hat{Att}_{ij}^{Position}=E_{ij}W_{p} \tag{6}\]We then combine the positional attention and the content(dot-product) attention scores and apply row-wise normalization. \[\hat{Att}^{Content}_{ij}=\frac{QK^{T}}{\sqrt{d}}_{ij} \tag{7}\] \[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention. As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism. Self-Virtual-Edges, \(E_{ii}\), are Virtual Edges between a node to itself. They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Self-walks were shown (Dwivedi et al. (2022)) to be an effective positional encoding. Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column. Throughout our experiments, we take an existing Graph Transformer architecture and use it "as is" as our Transformer module without doing any hyper-parameter search. That is, we take a graph Transformer with the same hyperparameters used by the original work and only add our positional attention and encoding.2

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

Our architecture integrates with the Transformer and consequently suffers from the quadratic memory complexity of the attention mechanism, which can restrict its applicability to larger graphs.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix.

We use GatedGCN as GNN modules, and four attention heads in Transformer modules. Transformer models use six layers, \(GNN\to Transformer\) uses 3 GNN layers and 3 Transformer layers.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim, evidence, and reviewer comments, I'll evaluate whether "We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)" is well-stated or overstated.

The evaluation reveals some concerns about this claim:

1. While the architecture does incorporate structural data into Transformers, calling it "simple" may be overstating given:
- The method involves multiple complex components (virtual edges, positional attention, content attention)
- The reviewer points out some parts are "confusing" and need clarification
- The first step of learning a dense adjacency matrix is noted as potentially "very expensive" for large graphs

2. The paper provides detailed technical descriptions and experimental validation, but the simplicity claim isn't strongly supported when compared to existing approaches.

<score>3</score>

The claim deserves a middle score of 3 - while the method appears effective and is well-documented, characterizing it as "simple" may overstate its accessibility and implementation complexity relative to other approaches.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD).

The review comment to be evaluated is:

Strength:
1. The proposed method performs favorably on benchmarks datasets.
2. The method is effective but is also quite simple in term of implementing it.
3. The counter example of the 2-d grid histogram counting problem for graph transformer is interesting as all state of the art model fail to solve it.

Weakness:
1. The technical contribution is incremental. Graph diffusion (both in discrete and the ODE form) has been studied and applied in the domain of graph neutral network. While adding graph diffusion to the positional attention is new, it is a trivial extension. [1][2][3]
2. While it is surprising that most graph transformer could not solve the grid based counting problem, this problem itself in the context of graph can be easily solved by diffusion matrix itself if node color is expressed as node feature with one-hot encoding.

[1] https://papers.nips.cc/paper/2019/hash/23c894276a2c5a16470e6a31f4618d73-Abstract.html
[2] https://aclanthology.org/2021.emnlp-main.642.pdf
[3] https://openreview.net/forum?id=EMxu-dzvJk

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

We now describe our approach for taking a graph \(G=(X,A)\) with a node embeddings matrix \(X\) and edges \(A\) and processing it using the Transformer, which usually takes only a single matrix \(X\) as input. Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

We stack different powers of \(A\) into a 3 dimensional tensor \(E\in R^{n\times n\times k}\) \[E=[I|A|A^{2}|..|A^{k}] \tag{1}\] where \(k\) is the number of stacks and \(|\) denotes stacking matrices along the 3ed dimension and \(I\) is the identity matrix.

Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network. Each layer consists of 1 hidden layer with batch norm, ReLU activation, and a residual connection. \[Edge\text{-}FFN(E_{ij})=ReLU(BN(E_{ij}W1))W2^{1} \tag{2}\] \[E_{ij}=Edge\text{-}FFN(E_{ij})+E_{ij} \tag{3}\]

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself. Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\). \[\hat{Att}_{ij}^{Position}=E_{ij}W_{p} \tag{6}\]We then combine the positional attention and the content(dot-product) attention scores and apply row-wise normalization. \[\hat{Att}^{Content}_{ij}=\frac{QK^{T}}{\sqrt{d}}_{ij} \tag{7}\] \[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention. As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism. Self-Virtual-Edges, \(E_{ii}\), are Virtual Edges between a node to itself. They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Self-walks were shown (Dwivedi et al. (2022)) to be an effective positional encoding. Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column. Throughout our experiments, we take an existing Graph Transformer architecture and use it "as is" as our Transformer module without doing any hyper-parameter search. That is, we take a graph Transformer with the same hyperparameters used by the original work and only add our positional attention and encoding.2

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

Our architecture integrates with the Transformer and consequently suffers from the quadratic memory complexity of the attention mechanism, which can restrict its applicability to larger graphs.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix.

We use GatedGCN as GNN modules, and four attention heads in Transformer modules. Transformer models use six layers, \(GNN\to Transformer\) uses 3 GNN layers and 3 Transformer layers.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim "We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)" and the evidence provided:

The claim appears to be moderately overstated. While GD does provide a way to incorporate structural data into Transformers, calling it "simple" is questionable given:
1. The architecture involves multiple complex components (Virtual Edges, Edge-FFN, positional/content attention combination)
2. The reviewer points out that graph diffusion has been previously studied, and this is an incremental extension
3. The technical details and equations presented show considerable complexity in implementation

However, the method is relatively straightforward compared to some alternatives and achieves strong results with minimal tuning, partially supporting the "simple" characterization.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD).

The review comment to be evaluated is:

Pros:
The experiments can effectively show the performance of the proposed method.

Cons:
1. There are existing works [1, 2] about incorporating graph diffusion in attention-based GNN that are not covered in this work. Besides, in the Related Work section, the subsection "Diffusion" might be revised to "Diffusion in Graph" to avoid any ambiguity with "Diffusion Models".
2. The improvement on many benchmarks is limited compared to the standard deviations.
3. The models in Table 1 are not well described in the main context. For example, the settings of "GNN →Transformer" and "GNN | Transformer+RWPE" can be briefly mentioned in 5.1.1. Moreover, the necessity of designing "GNN →Transformer" and "GNN | Transformer+RWPE" for comparison is not clear.
4. About the figure drawing: Figure 1 is a little hard to understand since it contains terms like "Edge-Wise FFN/projection" and unclear attention maps that are only described in the latter sections. The Transformer architecture in Figure 3 is not well drawn. The Positional Attention module is even partly outside the Multi-Head Attention module.

[1] Wang, Guangtao, Rex Ying, Jing Huang, and Jure Leskovec. "Multi-hop attention graph neural network." arXiv preprint arXiv:2009.14332 (2020).

[2] Liu, Yonghao, Renchu Guan, Fausto Giunchiglia, Yanchun Liang, and Xiaoyue Feng. "Deep attention diffusion graph neural networks for text classification." In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8142-8152. 2021.

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

We now describe our approach for taking a graph \(G=(X,A)\) with a node embeddings matrix \(X\) and edges \(A\) and processing it using the Transformer, which usually takes only a single matrix \(X\) as input. Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

We stack different powers of \(A\) into a 3 dimensional tensor \(E\in R^{n\times n\times k}\) \[E=[I|A|A^{2}|..|A^{k}] \tag{1}\] where \(k\) is the number of stacks and \(|\) denotes stacking matrices along the 3ed dimension and \(I\) is the identity matrix.

Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network. Each layer consists of 1 hidden layer with batch norm, ReLU activation, and a residual connection. \[Edge\text{-}FFN(E_{ij})=ReLU(BN(E_{ij}W1))W2^{1} \tag{2}\] \[E_{ij}=Edge\text{-}FFN(E_{ij})+E_{ij} \tag{3}\]

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself. Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\). \[\hat{Att}_{ij}^{Position}=E_{ij}W_{p} \tag{6}\]We then combine the positional attention and the content(dot-product) attention scores and apply row-wise normalization. \[\hat{Att}^{Content}_{ij}=\frac{QK^{T}}{\sqrt{d}}_{ij} \tag{7}\] \[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention. As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism. Self-Virtual-Edges, \(E_{ii}\), are Virtual Edges between a node to itself. They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Self-walks were shown (Dwivedi et al. (2022)) to be an effective positional encoding. Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column. Throughout our experiments, we take an existing Graph Transformer architecture and use it "as is" as our Transformer module without doing any hyper-parameter search. That is, we take a graph Transformer with the same hyperparameters used by the original work and only add our positional attention and encoding.2

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

Our architecture integrates with the Transformer and consequently suffers from the quadratic memory complexity of the attention mechanism, which can restrict its applicability to larger graphs.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix.

We use GatedGCN as GNN modules, and four attention heads in Transformer modules. Transformer models use six layers, \(GNN\to Transformer\) uses 3 GNN layers and 3 Transformer layers.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim "We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)" and the evidence provided, I'll assess its validity:

The claim deserves scrutiny particularly regarding the word "simple." While the architecture builds on existing concepts, it involves multiple complex components:
1. Learning new adjacency matrices using node/edge features
2. Stacking different powers of adjacency matrices
3. Edge-wise feed-forward networks
4. Combining positional and content attention mechanisms

The reviewer points out that some architectural elements aren't well described and Figure 1 is "hard to understand." The drawing issues in Figure 3 also suggest the architecture may not be as simple as claimed.

However, the claim is not severely overstated since the paper:
- Thoroughly describes the methodology
- Acknowledges limitations
- Shows implementation details
- Demonstrates practical effectiveness

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD).

The review comment to be evaluated is:

Strength:
1. The paper presents two novel design for a graph transformer, the graph diffuser and context attention.
2. The paper provides many experiments in different datasets to demonstrate the performance of their approach.
3. The detail description of the experimental setup is also useful in understanding the benchmarks.

Weakness:
1. The authors should add more literature review. By googling "graph transformer", there is a survey [1] on arXiv that mentioned various methods regarding graph transformer. But some of those methods are not reviewed by the authors. Also, the experiments should include more baselines.
2. When use weighted adjacency, the model does not leverage any information from original adjacency.
3. More ablation study are needed to reveal the importance of each component. For example, context attention only, context attention + weighted adj, context attention + diffuser + weighted adj, and so on.

[1] Min, E., Chen, R., Bian, Y., Xu, T., Zhao, K., Huang, W., ... & Rong, Y. (2022). Transformer for Graphs: An Overview from Architecture Perspective. arXiv preprint arXiv:2202.08455. (https://arxiv.org/abs/2202.08455)

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

We now describe our approach for taking a graph \(G=(X,A)\) with a node embeddings matrix \(X\) and edges \(A\) and processing it using the Transformer, which usually takes only a single matrix \(X\) as input. Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

We stack different powers of \(A\) into a 3 dimensional tensor \(E\in R^{n\times n\times k}\) \[E=[I|A|A^{2}|..|A^{k}] \tag{1}\] where \(k\) is the number of stacks and \(|\) denotes stacking matrices along the 3ed dimension and \(I\) is the identity matrix.

Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network. Each layer consists of 1 hidden layer with batch norm, ReLU activation, and a residual connection. \[Edge\text{-}FFN(E_{ij})=ReLU(BN(E_{ij}W1))W2^{1} \tag{2}\] \[E_{ij}=Edge\text{-}FFN(E_{ij})+E_{ij} \tag{3}\]

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself. Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\). \[\hat{Att}_{ij}^{Position}=E_{ij}W_{p} \tag{6}\]We then combine the positional attention and the content(dot-product) attention scores and apply row-wise normalization. \[\hat{Att}^{Content}_{ij}=\frac{QK^{T}}{\sqrt{d}}_{ij} \tag{7}\] \[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention. As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism. Self-Virtual-Edges, \(E_{ii}\), are Virtual Edges between a node to itself. They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Self-walks were shown (Dwivedi et al. (2022)) to be an effective positional encoding. Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column. Throughout our experiments, we take an existing Graph Transformer architecture and use it "as is" as our Transformer module without doing any hyper-parameter search. That is, we take a graph Transformer with the same hyperparameters used by the original work and only add our positional attention and encoding.2

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

Our architecture integrates with the Transformer and consequently suffers from the quadratic memory complexity of the attention mechanism, which can restrict its applicability to larger graphs.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix.

We use GatedGCN as GNN modules, and four attention heads in Transformer modules. Transformer models use six layers, \(GNN\to Transformer\) uses 3 GNN layers and 3 Transformer layers.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)" requires careful evaluation.

While GD does present an architecture for incorporating graph structure into Transformers, calling it "simple" is somewhat debatable. The evidence shows multiple components and mathematical formulations, including:
- Virtual Edges creation through matrix stacking
- Edge-wise FFN processing
- Learned adjacency matrices
- Complex attention mechanisms combining positional and content attention
- Self-Virtual Edges as positional encodings

The reviewer comments also note the need for more ablation studies to understand component interactions, suggesting the architecture may not be as simple as claimed. However, the authors do provide clear visualizations and explanations of the approach.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The intuition that guides us is that while the _aggregation_ scheme of GNNs is limited, thepropagation_ of information along the graph structure provides a valuable inductive bias for learning on graphs.

The evidence to be evaluated is:MP-GNNs Gori et al. (2005); Scarselli et al. (2008) have been the predominant method for graph representation learning in recent years, and have been applied to a wide variety of domains (e.g., Kosaraju et al. (2019); Nathani et al. (2019); Wang et al. (2019); Huang and Carley (2019); Yang et al. (2020); Ma et al. (2020); Wu et al. (2020); Zhang et al. (2020)), MP-GNNs update node representation by stacking multiple layers in which each node aggregates information from its local neighborhoodLi et al. (2016); Kipf and Welling (2017); Velickovic et al. (2018); Wu et al. (2019); Xu et al. (2019); Hamilton et al. (2017); Xu et al. (2019). However, as mentioned above, they suffer from under-reaching Barcelo et al. (2020), over-smoothing Wu et al. (2020) and over-squashing Alon and Yahav (2021); Topping et al. (2022).

Another line of work uses the attention mechanism Velickovic et al. (2018); Brody et al. (2022) to dynamically propagate information in the graph rather than use the original adjacency matrix or Laplacian.

Others combine Transformer with local MP-GNN models by interleaving Rampasek et al. (2022) or stacking them Jain et al. (2021), much in the same way the Transformers were stacked on top of CNNs in computer vision Carion et al. (2020). Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph. SAN Kreuzer et al. (2021) significantly improved this idea by using an invariant aggregation for the eigenvectors. Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Since then, various works(e.g SAT Chen et al. (2022), EGT Hussain et al. (2021), GRPE Park et al. (2022)) applied the graph structure as a soft bias to the attention.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges.

_Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates. In a row-normalized adjacency matrix \(A\), \(A_{ij}^{k}\) corresponds to the probability of getting from node \(i\) to node \(j\) in a \(k\) step random walk.

Multiplication by the adjacency matrix is closely related to the aggregation scheme of MP-GNNS Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes.

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be due to the over-squashing phenomenon. Indeed in a \(10\times 10\) grid, passing information between opposing nodes in the same row will require \(9\) message passing layers to deliver a single message that has been "squashed" with nearly 90k other messages.

Another area of interest is amplifying virtual edges, for example, by considering paths between nodes rather than just information propagation.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim appears well-stated (score of 1) based on the extensive evidence provided. The evidence thoroughly demonstrates the limitations of GNN aggregation while highlighting the value of graph structure propagation:

1. The text provides detailed examples of how information propagation along graph structures has been successfully leveraged in multiple approaches (Graph Transformers, Virtual Edges, etc.)

2. The evidence acknowledges known limitations of message-passing GNNs (under-reaching, over-smoothing, over-squashing) while showing how propagation-based methods help address these issues

3. The figures clearly illustrate how structural information propagation works through Virtual Edges and attention mechanisms

4. The evidence presents a balanced view, discussing both benefits and challenges of different propagation approaches without overstating their capabilities

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The intuition that guides us is that while the _aggregation_ scheme of GNNs is limited, thepropagation_ of information along the graph structure provides a valuable inductive bias for learning on graphs.

The review comment to be evaluated is:

Strength
- The method is motivated by the failure of current architectures over a synthetic task. The proposed method brings significant improvement to it.
-  The method demonstrates empirical gains over some datasets, mostly on ones that requires capturing long-range interactions

Weaknesses
- Some important parts of the paper are confusing, mostly due to the ordering of the sections and the phrases:
    - It is not clear to me what "learn to combine information propagation over multiple different propagation steps in an
end-to-end manner" means. What are the "multiple different propagation steps"?
    - The use of the phrase "virtual edge" is somewhat confusing. Based on the later formulas, it seems these virtual edges are essentially generated features between all pair of nodes (based on 4.1), and replace the original edges/edge features. But the phrase itself and Figure 1 makes it seem like the virtual edges are added to the original set of edges.
- Some of the statements seem to be over-claimed:
    - "However, given the arbitrary structure of graphs, incorporating the input into the Transformer remains a challenging aspect in designing GTs, and so far, there has been no universal solution. We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)..." It should be clarified that a lot of the current graph transformers designs are expressive enough incorporate arbitrary structure of graphs (e.g. Graphormer, GRAPHGPS, etc) and showed strong results. The authors should narrow down the claim to improving the modeling of long-range interaction.
    - "this work is the first Graph Transformer to [...] learn to combine information propagation over multiple different propagation steps in an end-to-end manner." It is not particularly clear to me what this statement means, but seems this is referring to feeding stacked random walk matrices through feed-forward neural networks in 4.1.2. I believe this has already been mentioned in GRAPHGPS and GraphiT (i.e. Relative PE using random walks).
- The first step of learning a dense adjacency matrix could be very expensive for large graphs.
- The empirical gain on ZINC and most of the OGB datasets seem quite small compared to the variance.

The evidence to be evaluated is:MP-GNNs Gori et al. (2005); Scarselli et al. (2008) have been the predominant method for graph representation learning in recent years, and have been applied to a wide variety of domains (e.g., Kosaraju et al. (2019); Nathani et al. (2019); Wang et al. (2019); Huang and Carley (2019); Yang et al. (2020); Ma et al. (2020); Wu et al. (2020); Zhang et al. (2020)), MP-GNNs update node representation by stacking multiple layers in which each node aggregates information from its local neighborhoodLi et al. (2016); Kipf and Welling (2017); Velickovic et al. (2018); Wu et al. (2019); Xu et al. (2019); Hamilton et al. (2017); Xu et al. (2019). However, as mentioned above, they suffer from under-reaching Barcelo et al. (2020), over-smoothing Wu et al. (2020) and over-squashing Alon and Yahav (2021); Topping et al. (2022).

Another line of work uses the attention mechanism Velickovic et al. (2018); Brody et al. (2022) to dynamically propagate information in the graph rather than use the original adjacency matrix or Laplacian.

Others combine Transformer with local MP-GNN models by interleaving Rampasek et al. (2022) or stacking them Jain et al. (2021), much in the same way the Transformers were stacked on top of CNNs in computer vision Carion et al. (2020). Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph. SAN Kreuzer et al. (2021) significantly improved this idea by using an invariant aggregation for the eigenvectors. Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Since then, various works(e.g SAT Chen et al. (2022), EGT Hussain et al. (2021), GRPE Park et al. (2022)) applied the graph structure as a soft bias to the attention.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges.

_Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates. In a row-normalized adjacency matrix \(A\), \(A_{ij}^{k}\) corresponds to the probability of getting from node \(i\) to node \(j\) in a \(k\) step random walk.

Multiplication by the adjacency matrix is closely related to the aggregation scheme of MP-GNNS Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes.

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be due to the over-squashing phenomenon. Indeed in a \(10\times 10\) grid, passing information between opposing nodes in the same row will require \(9\) message passing layers to deliver a single message that has been "squashed" with nearly 90k other messages.

Another area of interest is amplifying virtual edges, for example, by considering paths between nodes rather than just information propagation.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim, evidence, and review comments, here is my evaluation:

The claim that GNN aggregation is limited while propagation provides valuable inductive bias appears moderately well-supported but somewhat oversimplified. The evidence demonstrates clear limitations of GNN aggregation (under-reaching, over-smoothing, over-squashing) and shows benefits of propagation through virtual edges and multi-step information flow. However, the review points out that other approaches like Graphormer and GraphGPS already incorporate similar propagation concepts. The claim could be more precise in acknowledging existing solutions rather than implying this is a novel insight.

<score>3</score>

Justification:
1. The evidence supports basic premise about GNN limitations through citations showing under-reaching, over-smoothing issues
2. The review correctly notes that several existing architectures already leverage propagation effectively
3. The claim would be stronger if it:
- Acknowledged existing solutions in the propagation space
- More precisely defined what aspects of propagation provide the valuable inductive bias
- Clarified the specific limitations in aggregation being referenced
4. The figures and technical details about virtual edges and information flow support the general premise but don't fully justify the broad nature of the claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The intuition that guides us is that while the _aggregation_ scheme of GNNs is limited, thepropagation_ of information along the graph structure provides a valuable inductive bias for learning on graphs.

The review comment to be evaluated is:

Strength:
1. The proposed method performs favorably on benchmarks datasets.
2. The method is effective but is also quite simple in term of implementing it.
3. The counter example of the 2-d grid histogram counting problem for graph transformer is interesting as all state of the art model fail to solve it.

Weakness:
1. The technical contribution is incremental. Graph diffusion (both in discrete and the ODE form) has been studied and applied in the domain of graph neutral network. While adding graph diffusion to the positional attention is new, it is a trivial extension. [1][2][3]
2. While it is surprising that most graph transformer could not solve the grid based counting problem, this problem itself in the context of graph can be easily solved by diffusion matrix itself if node color is expressed as node feature with one-hot encoding.

[1] https://papers.nips.cc/paper/2019/hash/23c894276a2c5a16470e6a31f4618d73-Abstract.html
[2] https://aclanthology.org/2021.emnlp-main.642.pdf
[3] https://openreview.net/forum?id=EMxu-dzvJk

The evidence to be evaluated is:MP-GNNs Gori et al. (2005); Scarselli et al. (2008) have been the predominant method for graph representation learning in recent years, and have been applied to a wide variety of domains (e.g., Kosaraju et al. (2019); Nathani et al. (2019); Wang et al. (2019); Huang and Carley (2019); Yang et al. (2020); Ma et al. (2020); Wu et al. (2020); Zhang et al. (2020)), MP-GNNs update node representation by stacking multiple layers in which each node aggregates information from its local neighborhoodLi et al. (2016); Kipf and Welling (2017); Velickovic et al. (2018); Wu et al. (2019); Xu et al. (2019); Hamilton et al. (2017); Xu et al. (2019). However, as mentioned above, they suffer from under-reaching Barcelo et al. (2020), over-smoothing Wu et al. (2020) and over-squashing Alon and Yahav (2021); Topping et al. (2022).

Another line of work uses the attention mechanism Velickovic et al. (2018); Brody et al. (2022) to dynamically propagate information in the graph rather than use the original adjacency matrix or Laplacian.

Others combine Transformer with local MP-GNN models by interleaving Rampasek et al. (2022) or stacking them Jain et al. (2021), much in the same way the Transformers were stacked on top of CNNs in computer vision Carion et al. (2020). Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph. SAN Kreuzer et al. (2021) significantly improved this idea by using an invariant aggregation for the eigenvectors. Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Since then, various works(e.g SAT Chen et al. (2022), EGT Hussain et al. (2021), GRPE Park et al. (2022)) applied the graph structure as a soft bias to the attention.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges.

_Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates. In a row-normalized adjacency matrix \(A\), \(A_{ij}^{k}\) corresponds to the probability of getting from node \(i\) to node \(j\) in a \(k\) step random walk.

Multiplication by the adjacency matrix is closely related to the aggregation scheme of MP-GNNS Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes.

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be due to the over-squashing phenomenon. Indeed in a \(10\times 10\) grid, passing information between opposing nodes in the same row will require \(9\) message passing layers to deliver a single message that has been "squashed" with nearly 90k other messages.

Another area of interest is amplifying virtual edges, for example, by considering paths between nodes rather than just information propagation.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim, evidence, and reviewer comments, I'll assess whether the statement is well-stated or overstated.

Summary of Evaluation:
The claim suggests that while GNN aggregation is limited, graph propagation provides valuable inductive bias. The evidence strongly supports this through:
1. Extensive discussion of how different propagation methods (virtual edges, multi-step diffusion) overcome traditional GNN limitations
2. Demonstration through figures showing how information propagates across multiple steps
3. Specific examples of structural information captured through propagation
4. Reviewer comments acknowledge the effectiveness of graph diffusion, though noting it's not entirely novel

The claim is conservatively stated and well-supported by the evidence, carefully avoiding overstatement of the benefits while acknowledging the limitations of current approaches.

<score>1</score>

Justification:
1. The claim uses measured language ("intuition that guides us") rather than definitive statements
2. Evidence directly shows how propagation provides structural information (e.g., odd length cycles, node degrees)
3. The figures demonstrate concrete examples of information propagation
4. Even critical reviewer comments acknowledge the effectiveness of the approach
5. The claim doesn't overstate novelty or superiority over other methods


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The intuition that guides us is that while the _aggregation_ scheme of GNNs is limited, thepropagation_ of information along the graph structure provides a valuable inductive bias for learning on graphs.

The review comment to be evaluated is:

Pros:
The experiments can effectively show the performance of the proposed method.

Cons:
1. There are existing works [1, 2] about incorporating graph diffusion in attention-based GNN that are not covered in this work. Besides, in the Related Work section, the subsection "Diffusion" might be revised to "Diffusion in Graph" to avoid any ambiguity with "Diffusion Models".
2. The improvement on many benchmarks is limited compared to the standard deviations.
3. The models in Table 1 are not well described in the main context. For example, the settings of "GNN →Transformer" and "GNN | Transformer+RWPE" can be briefly mentioned in 5.1.1. Moreover, the necessity of designing "GNN →Transformer" and "GNN | Transformer+RWPE" for comparison is not clear.
4. About the figure drawing: Figure 1 is a little hard to understand since it contains terms like "Edge-Wise FFN/projection" and unclear attention maps that are only described in the latter sections. The Transformer architecture in Figure 3 is not well drawn. The Positional Attention module is even partly outside the Multi-Head Attention module.

[1] Wang, Guangtao, Rex Ying, Jing Huang, and Jure Leskovec. "Multi-hop attention graph neural network." arXiv preprint arXiv:2009.14332 (2020).

[2] Liu, Yonghao, Renchu Guan, Fausto Giunchiglia, Yanchun Liang, and Xiaoyue Feng. "Deep attention diffusion graph neural networks for text classification." In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8142-8152. 2021.

The evidence to be evaluated is:MP-GNNs Gori et al. (2005); Scarselli et al. (2008) have been the predominant method for graph representation learning in recent years, and have been applied to a wide variety of domains (e.g., Kosaraju et al. (2019); Nathani et al. (2019); Wang et al. (2019); Huang and Carley (2019); Yang et al. (2020); Ma et al. (2020); Wu et al. (2020); Zhang et al. (2020)), MP-GNNs update node representation by stacking multiple layers in which each node aggregates information from its local neighborhoodLi et al. (2016); Kipf and Welling (2017); Velickovic et al. (2018); Wu et al. (2019); Xu et al. (2019); Hamilton et al. (2017); Xu et al. (2019). However, as mentioned above, they suffer from under-reaching Barcelo et al. (2020), over-smoothing Wu et al. (2020) and over-squashing Alon and Yahav (2021); Topping et al. (2022).

Another line of work uses the attention mechanism Velickovic et al. (2018); Brody et al. (2022) to dynamically propagate information in the graph rather than use the original adjacency matrix or Laplacian.

Others combine Transformer with local MP-GNN models by interleaving Rampasek et al. (2022) or stacking them Jain et al. (2021), much in the same way the Transformers were stacked on top of CNNs in computer vision Carion et al. (2020). Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph. SAN Kreuzer et al. (2021) significantly improved this idea by using an invariant aggregation for the eigenvectors. Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Since then, various works(e.g SAT Chen et al. (2022), EGT Hussain et al. (2021), GRPE Park et al. (2022)) applied the graph structure as a soft bias to the attention.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges.

_Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates. In a row-normalized adjacency matrix \(A\), \(A_{ij}^{k}\) corresponds to the probability of getting from node \(i\) to node \(j\) in a \(k\) step random walk.

Multiplication by the adjacency matrix is closely related to the aggregation scheme of MP-GNNS Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes.

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be due to the over-squashing phenomenon. Indeed in a \(10\times 10\) grid, passing information between opposing nodes in the same row will require \(9\) message passing layers to deliver a single message that has been "squashed" with nearly 90k other messages.

Another area of interest is amplifying virtual edges, for example, by considering paths between nodes rather than just information propagation.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the assertion about GNN aggregation limitations and propagation's value to be fairly well-supported, but with some overstatement.

The evidence demonstrates clear limitations of GNN aggregation through documented issues like "under-reaching," "over-smoothing," and "over-squashing." The value of propagation is supported by discussions of Virtual Edges and information flow across multiple propagation steps. However, the claim makes a broad intuitive statement without acknowledging ongoing work addressing aggregation limitations or potential drawbacks of propagation approaches.

The review comments highlight limited improvements in benchmarks, suggesting the benefits may not be as definitive as the claim implies.

<score>3</score>

Justification:
1. Evidence supports basic premise but claim overstates certainty
2. Doesn't acknowledge existing solutions or limitations
3. Review points out limited benchmark improvements
4. Technical explanation in figures/text provides qualified support
5. "Intuition" framing makes broad generalization without necessary caveats


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The intuition that guides us is that while the _aggregation_ scheme of GNNs is limited, thepropagation_ of information along the graph structure provides a valuable inductive bias for learning on graphs.

The review comment to be evaluated is:

Strength:
1. The paper presents two novel design for a graph transformer, the graph diffuser and context attention.
2. The paper provides many experiments in different datasets to demonstrate the performance of their approach.
3. The detail description of the experimental setup is also useful in understanding the benchmarks.

Weakness:
1. The authors should add more literature review. By googling "graph transformer", there is a survey [1] on arXiv that mentioned various methods regarding graph transformer. But some of those methods are not reviewed by the authors. Also, the experiments should include more baselines.
2. When use weighted adjacency, the model does not leverage any information from original adjacency.
3. More ablation study are needed to reveal the importance of each component. For example, context attention only, context attention + weighted adj, context attention + diffuser + weighted adj, and so on.

[1] Min, E., Chen, R., Bian, Y., Xu, T., Zhao, K., Huang, W., ... & Rong, Y. (2022). Transformer for Graphs: An Overview from Architecture Perspective. arXiv preprint arXiv:2202.08455. (https://arxiv.org/abs/2202.08455)

The evidence to be evaluated is:MP-GNNs Gori et al. (2005); Scarselli et al. (2008) have been the predominant method for graph representation learning in recent years, and have been applied to a wide variety of domains (e.g., Kosaraju et al. (2019); Nathani et al. (2019); Wang et al. (2019); Huang and Carley (2019); Yang et al. (2020); Ma et al. (2020); Wu et al. (2020); Zhang et al. (2020)), MP-GNNs update node representation by stacking multiple layers in which each node aggregates information from its local neighborhoodLi et al. (2016); Kipf and Welling (2017); Velickovic et al. (2018); Wu et al. (2019); Xu et al. (2019); Hamilton et al. (2017); Xu et al. (2019). However, as mentioned above, they suffer from under-reaching Barcelo et al. (2020), over-smoothing Wu et al. (2020) and over-squashing Alon and Yahav (2021); Topping et al. (2022).

Another line of work uses the attention mechanism Velickovic et al. (2018); Brody et al. (2022) to dynamically propagate information in the graph rather than use the original adjacency matrix or Laplacian.

Others combine Transformer with local MP-GNN models by interleaving Rampasek et al. (2022) or stacking them Jain et al. (2021), much in the same way the Transformers were stacked on top of CNNs in computer vision Carion et al. (2020). Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph. SAN Kreuzer et al. (2021) significantly improved this idea by using an invariant aggregation for the eigenvectors. Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Since then, various works(e.g SAT Chen et al. (2022), EGT Hussain et al. (2021), GRPE Park et al. (2022)) applied the graph structure as a soft bias to the attention.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges.

_Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates. In a row-normalized adjacency matrix \(A\), \(A_{ij}^{k}\) corresponds to the probability of getting from node \(i\) to node \(j\) in a \(k\) step random walk.

Multiplication by the adjacency matrix is closely related to the aggregation scheme of MP-GNNS Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes.

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be due to the over-squashing phenomenon. Indeed in a \(10\times 10\) grid, passing information between opposing nodes in the same row will require \(9\) message passing layers to deliver a single message that has been "squashed" with nearly 90k other messages.

Another area of interest is amplifying virtual edges, for example, by considering paths between nodes rather than just information propagation.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim, evidence, and review comments, I'll evaluate whether the statement about GNN aggregation limitations and propagation benefits is well-stated or overstated.

Summary:
The evidence strongly supports this claim through multiple angles:
1. The paper extensively discusses limitations of traditional GNN aggregation (under-reaching, over-smoothing, over-squashing)
2. The evidence demonstrates the value of information propagation through virtual edges and multi-step diffusion
3. The illustrations and technical details show how propagation along graph structure provides meaningful structural information and improved learning capabilities
4. The review comments, while noting some methodological gaps, don't contradict this core claim

The claim is carefully worded with "intuition that guides us" rather than making absolute statements, and the term "inductive bias" appropriately characterizes the role of graph structure propagation.

<score>1</score>

Justification:
1. The claim is well-stated as it accurately reflects the evidence without exaggeration
2. The wording acknowledges limitations ("limited") while highlighting benefits in measured terms
3. The extensive experimental evidence and theoretical framework presented support both aspects of the claim
4. The claim aligns with the paper's demonstrated improvements through novel propagation methods
5. No overreaching conclusions or unsupported assertions are made