Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose _Graph Diffuser_ (GD) to address this challenge.

The review comment to be evaluated is:

Strength
- The method is motivated by the failure of current architectures over a synthetic task. The proposed method brings significant improvement to it.
-  The method demonstrates empirical gains over some datasets, mostly on ones that requires capturing long-range interactions

Weaknesses
- Some important parts of the paper are confusing, mostly due to the ordering of the sections and the phrases:
    - It is not clear to me what "learn to combine information propagation over multiple different propagation steps in an
end-to-end manner" means. What are the "multiple different propagation steps"?
    - The use of the phrase "virtual edge" is somewhat confusing. Based on the later formulas, it seems these virtual edges are essentially generated features between all pair of nodes (based on 4.1), and replace the original edges/edge features. But the phrase itself and Figure 1 makes it seem like the virtual edges are added to the original set of edges.
- Some of the statements seem to be over-claimed:
    - "However, given the arbitrary structure of graphs, incorporating the input into the Transformer remains a challenging aspect in designing GTs, and so far, there has been no universal solution. We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)..." It should be clarified that a lot of the current graph transformers designs are expressive enough incorporate arbitrary structure of graphs (e.g. Graphormer, GRAPHGPS, etc) and showed strong results. The authors should narrow down the claim to improving the modeling of long-range interaction.
    - "this work is the first Graph Transformer to [...] learn to combine information propagation over multiple different propagation steps in an end-to-end manner." It is not particularly clear to me what this statement means, but seems this is referring to feeding stacked random walk matrices through feed-forward neural networks in 4.1.2. I believe this has already been mentioned in GRAPHGPS and GraphiT (i.e. Relative PE using random walks).
- The first step of learning a dense adjacency matrix could be very expensive for large graphs.
- The empirical gain on ZINC and most of the OGB datasets seem quite small compared to the variance.

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships.

Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix. Moreover, we will release our code as open-source with clear instructions on how to reproduce all of our results, as well as how to extend our model and apply it to new datasets.

For Graph Difffuser, we found 3 layers sufficient.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

For each variation, we evaluate the unmodified architecture with no positional embedding or positional attention, the architecture with our positional encoding and attention added, but not learning the adjacency matrix(+ _Graph Diffuser_), and one we learn a new weighted adjacency matrix(+ _Weighted Adjacency_), as described in Section 4.1.3.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, the claim "We propose Graph Diffuser (GD) to address this challenge" appears somewhat overstated when considered alongside the evidence and reviewer comments. While the paper does present Graph Diffuser as a novel approach with some demonstrated improvements, the reviewer points out that similar capabilities exist in other graph transformers (e.g., Graphormer, GRAPHGPS). The evidence shows GD combines multiple techniques in an interesting way, but claiming it uniquely addresses graph transformer challenges overstates its novelty and contribution, especially given the reviewer's note about small empirical gains on several datasets.

<score>4</score>

Justification:

1. Overstatement of uniqueness:
- The claim suggests GD is a universal solution, but reviewer notes "a lot of current graph transformers designs are expressive enough to incorporate arbitrary structure of graphs"
- The paper's claim about being "first" to combine multiple propagation steps is questioned by reviewer who points to similar approaches in GRAPHGPS and GraphiT

2. Limited evidence of superiority:
- Paper states "achieves new state-of-the-art on most datasets" but reviewer indicates "empirical gain on ZINC and most of the OGB datasets seem quite small compared to the variance"
- Evidence shows improvements mainly on tasks requiring long-range interactions

3. Practical limitations:
- Reviewer points out that "learning a dense adjacency matrix could be very expensive for large graphs" - an important limitation not acknowledged in the broad initial claim

4. Documentation quality:
- While the paper provides implementation details and promises code release, the reviewer notes confusion about key concepts and terminology (e.g., "virtual edge" definition)

The claim would be better stated by:
1. Specifying the particular challenges GD addresses (long-range interactions)
2. Acknowledging existing solutions while highlighting incremental improvements
3. Being more precise about performance gains
4. Noting computational limitations


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose _Graph Diffuser_ (GD) to address this challenge.

The review comment to be evaluated is:

Strength:
1. The proposed method performs favorably on benchmarks datasets.
2. The method is effective but is also quite simple in term of implementing it.
3. The counter example of the 2-d grid histogram counting problem for graph transformer is interesting as all state of the art model fail to solve it.

Weakness:
1. The technical contribution is incremental. Graph diffusion (both in discrete and the ODE form) has been studied and applied in the domain of graph neutral network. While adding graph diffusion to the positional attention is new, it is a trivial extension. [1][2][3]
2. While it is surprising that most graph transformer could not solve the grid based counting problem, this problem itself in the context of graph can be easily solved by diffusion matrix itself if node color is expressed as node feature with one-hot encoding.

[1] https://papers.nips.cc/paper/2019/hash/23c894276a2c5a16470e6a31f4618d73-Abstract.html
[2] https://aclanthology.org/2021.emnlp-main.642.pdf
[3] https://openreview.net/forum?id=EMxu-dzvJk

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships.

Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix. Moreover, we will release our code as open-source with clear instructions on how to reproduce all of our results, as well as how to extend our model and apply it to new datasets.

For Graph Difffuser, we found 3 layers sufficient.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

For each variation, we evaluate the unmodified architecture with no positional embedding or positional attention, the architecture with our positional encoding and attention added, but not learning the adjacency matrix(+ _Graph Diffuser_), and one we learn a new weighted adjacency matrix(+ _Weighted Adjacency_), as described in Section 4.1.3.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "We propose Graph Diffuser (GD) to address this challenge" considering the evidence and review comments:

Summary: The claim appears moderately overstated based on the available evidence and reviewer feedback. While the evidence shows Graph Diffuser is a functional and effective approach, the reviewer points out that the technical contribution is incremental, building upon existing graph diffusion concepts. The evidence demonstrates implementation details and positive results, but doesn't fully address the reviewer's point about technical novelty. The authors' claim of being "first" to learn adjacency matrices and combine multi-step propagation is challenged by the reviewer's citations of prior related work.

<score>3</score>

Justification:
1. The evidence shows GD is well-implemented and effective:
- Detailed architecture description with clear figures
- Demonstrated success on benchmarks
- Reproducible methodology

2. However, several factors suggest overstatement:
- Reviewer points out that graph diffusion is not new in this domain
- Citations [1][2][3] indicate similar prior work
- The claim of being "first" appears overstated given the reviewer's references
- Technical contribution appears to be an extension rather than fundamentally new approach

3. Mitigating factors:
- Authors provide comprehensive implementation details
- Results show practical effectiveness
- Novel combination of existing concepts for improved performance

The score of 3 reflects that while the method is valid and well-documented, the claim could better acknowledge its incremental nature and relationship to existing work.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose _Graph Diffuser_ (GD) to address this challenge.

The review comment to be evaluated is:

Pros:
The experiments can effectively show the performance of the proposed method.

Cons:
1. There are existing works [1, 2] about incorporating graph diffusion in attention-based GNN that are not covered in this work. Besides, in the Related Work section, the subsection "Diffusion" might be revised to "Diffusion in Graph" to avoid any ambiguity with "Diffusion Models".
2. The improvement on many benchmarks is limited compared to the standard deviations.
3. The models in Table 1 are not well described in the main context. For example, the settings of "GNN →Transformer" and "GNN | Transformer+RWPE" can be briefly mentioned in 5.1.1. Moreover, the necessity of designing "GNN →Transformer" and "GNN | Transformer+RWPE" for comparison is not clear.
4. About the figure drawing: Figure 1 is a little hard to understand since it contains terms like "Edge-Wise FFN/projection" and unclear attention maps that are only described in the latter sections. The Transformer architecture in Figure 3 is not well drawn. The Positional Attention module is even partly outside the Multi-Head Attention module.

[1] Wang, Guangtao, Rex Ying, Jing Huang, and Jure Leskovec. "Multi-hop attention graph neural network." arXiv preprint arXiv:2009.14332 (2020).

[2] Liu, Yonghao, Renchu Guan, Fausto Giunchiglia, Yanchun Liang, and Xiaoyue Feng. "Deep attention diffusion graph neural networks for text classification." In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8142-8152. 2021.

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships.

Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix. Moreover, we will release our code as open-source with clear instructions on how to reproduce all of our results, as well as how to extend our model and apply it to new datasets.

For Graph Difffuser, we found 3 layers sufficient.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

For each variation, we evaluate the unmodified architecture with no positional embedding or positional attention, the architecture with our positional encoding and attention added, but not learning the adjacency matrix(+ _Graph Diffuser_), and one we learn a new weighted adjacency matrix(+ _Weighted Adjacency_), as described in Section 4.1.3.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim "We propose Graph Diffuser (GD) to address this challenge" and the available evidence, I'll evaluate its validity:

Summary: While the evidence demonstrates that Graph Diffuser is indeed proposed and explained in detail with architectural diagrams and implementation details, the review comments raise important concerns about: 1) omission of related work on graph diffusion in attention-based GNNs, 2) limited improvements compared to standard deviations on benchmarks, and 3) unclear necessity of some comparative models. The claim itself is relatively modest and factual - simply stating they propose the method - without making grandiose claims about its capabilities or uniqueness.

<score>1</score>

Justification:
- The claim is well-stated and appropriately modest - it simply states they propose a method
- The evidence shows they did develop and detail the method thoroughly with:
  - Clear architectural diagrams (Figures 1 & 3)
  - Detailed explanation of components and implementation
  - Code release promised for reproducibility
  - Experimental validation
- While there are limitations noted in the review, these don't affect the validity of the basic claim of proposing the method
- The claim avoids overstating performance or novelty in its basic form


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose _Graph Diffuser_ (GD) to address this challenge.

The review comment to be evaluated is:

Strength:
1. The paper presents two novel design for a graph transformer, the graph diffuser and context attention.
2. The paper provides many experiments in different datasets to demonstrate the performance of their approach.
3. The detail description of the experimental setup is also useful in understanding the benchmarks.

Weakness:
1. The authors should add more literature review. By googling "graph transformer", there is a survey [1] on arXiv that mentioned various methods regarding graph transformer. But some of those methods are not reviewed by the authors. Also, the experiments should include more baselines.
2. When use weighted adjacency, the model does not leverage any information from original adjacency.
3. More ablation study are needed to reveal the importance of each component. For example, context attention only, context attention + weighted adj, context attention + diffuser + weighted adj, and so on.

[1] Min, E., Chen, R., Bian, Y., Xu, T., Zhao, K., Huang, W., ... & Rong, Y. (2022). Transformer for Graphs: An Overview from Architecture Perspective. arXiv preprint arXiv:2202.08455. (https://arxiv.org/abs/2202.08455)

The evidence to be evaluated is:In contrast, Graph Diffuser learns to combine multiple distances.

However, all of the works above use the original adjacency matrix or Laplacian to learn positional or relative encoding, unlike this work that learns the adjacency matrix using node and edge features.

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Our approach consists of 2 stages, illustrated in the left and right halves of Figure 3. First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships.

Graph Diffuser combines the 2 approaches. In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

As we will see in the next section, separating the attention based on content and positions and then combining them seems to assist with learning meaningful connections in the data and provides a natural visualization mechanism.

Finally, Graph Diffuser nearly solves the task entirely by combining both positional and content attention. Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

In this work, we introduced a simple and effective architecture for machine learning on graphs, Graph Diffuser. Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. With minimal hyperparameter tuning, Graph Diffuser achieves a new state-of-the-art on most datasets and reaches very competitive results on the rest.

We describe Graph Diffuser in detail in the text and figures and mention all relevant hyperparameters to reproduce our experiments in the appendix. Moreover, we will release our code as open-source with clear instructions on how to reproduce all of our results, as well as how to extend our model and apply it to new datasets.

For Graph Difffuser, we found 3 layers sufficient.

We Conducted a simplified experiment to see the effects of GD when used as an out-of-the-box addition to different Transformer compositions, with no hyperparameter tuning.

For each variation, we evaluate the unmodified architecture with no positional embedding or positional attention, the architecture with our positional encoding and attention added, but not learning the adjacency matrix(+ _Graph Diffuser_), and one we learn a new weighted adjacency matrix(+ _Weighted Adjacency_), as described in Section 4.1.3.

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "We propose Graph Diffuser (GD) to address this challenge."

1. Summary of evaluation:
The claim appears relatively well-stated, as the evidence demonstrates that Graph Diffuser is indeed proposed and thoroughly described with:
- Clear architectural details and illustrations (Figures 1 & 3)
- Specific technical contributions (learning adjacency matrix, combining multiple distances)
- Implementation details (3 layers, hyperparameters in appendix)
- Promised code release
However, the reviewer points out some limitations like incomplete literature review and need for more ablation studies, which suggests some aspects could be better substantiated.

<score>2</score>

Justification:
While the claim itself is straightforward, there are a few factors that prevent it from being perfectly well-stated (score 1):

1. Strengths shown in evidence:
- Detailed architectural description with figures
- Clear explanation of novel components
- Implementation details provided
- Code release promised
- Multiple experiments conducted

2. Limitations from review:
- Incomplete literature review
- Need for more baselines
- Limited ablation studies
- Questions about weighted adjacency usage

The claim could be slightly more precise by acknowledging it as one possible approach among others, but it's not significantly overstated since the basic proposition is well-supported by the detailed evidence provided.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation.

The evidence to be evaluated is:Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph.

Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Solving it requires far away nodes to communicate, and the communication should consider both the nodes' content (color) and relation within the graph (being in the same row or column).

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

\[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be because RWPE is unable to detect symmetries in the graph. Nodes 0,4,20 and 24 in the example illustrated in Figure 1(a) will all get the same embedding by RWPE.

Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

The average distance between nodes in this dataset is larger than that of any other dataset in our experiments, which can explain why Graph Transformers variations dominate its leaderboards. Since positional encoding was not found to be significant by the previous three highest ranking models in this dataset, and considering the large improvement of our model over GPS, it seems that positional attention brings significant benefits in this benchmark, which allows us to reach a new state-of-the-art.

As we can see in Table 5, adding our positional attention and encoding outperforms both RWSE and LapPE encodings.

Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. \begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Params.**} & \multicolumn{2}{c}{Peptides-func} & Peptides-struct \\ \cline{3-4}  & & **AP \(\uparrow\)** & **MAE \(\downarrow\)** \\ \hline GCN & 508k & 0.5930\(\pm\)0.0023 & 0.3496\(\pm\)0.0013 \\ GINE & 476k & 0.5498\(\pm\)0.0079 & 0.3547\(\pm\)0.0045 \\ GatedGCN & 509k & 0.5864\(\pm\)0.0077 & 0.3420\(\pm\)0.0013 \\ GatedGCN+RWSE & 506k & 0.6069\(\pm\)0.0035 & 0.3357\(\pm\)0.0006 \\ \hline Transformer+LapPE & 488k & 0.6326\(\pm\)0.0126 & 0.2529\(\pm\)0.0016 \\ SAN+LapPE & 493k & 0.6384\(\pm\)0.0121 & 0.2683\(\pm\)0.0043 \\ SAN+RWSE & 500k & 0.6439\(\pm\)0.0075 & 0.2545\(\pm\)0.0012 \\ \hline Graph Diffuser & 509k & **0.6651\(\pm\)0.001** & **0.2461\(\pm\)0.001** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim that "GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation" to be well-supported by the evidence. The paper provides:

1. Detailed mathematical formulations showing how structural relationships are learned (equations 4-9)
2. Empirical evidence through performance improvements (Table 5)
3. Visual demonstrations (Figure 1 and 3) showing how different attention mechanisms capture positional and content relationships
4. Clear experimental validation on the Peptides datasets where long-range interactions are important

The claim accurately reflects the capabilities demonstrated without overstating them.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation.

The review comment to be evaluated is:

Strength
- The method is motivated by the failure of current architectures over a synthetic task. The proposed method brings significant improvement to it.
-  The method demonstrates empirical gains over some datasets, mostly on ones that requires capturing long-range interactions

Weaknesses
- Some important parts of the paper are confusing, mostly due to the ordering of the sections and the phrases:
    - It is not clear to me what "learn to combine information propagation over multiple different propagation steps in an
end-to-end manner" means. What are the "multiple different propagation steps"?
    - The use of the phrase "virtual edge" is somewhat confusing. Based on the later formulas, it seems these virtual edges are essentially generated features between all pair of nodes (based on 4.1), and replace the original edges/edge features. But the phrase itself and Figure 1 makes it seem like the virtual edges are added to the original set of edges.
- Some of the statements seem to be over-claimed:
    - "However, given the arbitrary structure of graphs, incorporating the input into the Transformer remains a challenging aspect in designing GTs, and so far, there has been no universal solution. We propose a simple architecture for incorporating structural data into the Transformer, Graph Diffuser (GD)..." It should be clarified that a lot of the current graph transformers designs are expressive enough incorporate arbitrary structure of graphs (e.g. Graphormer, GRAPHGPS, etc) and showed strong results. The authors should narrow down the claim to improving the modeling of long-range interaction.
    - "this work is the first Graph Transformer to [...] learn to combine information propagation over multiple different propagation steps in an end-to-end manner." It is not particularly clear to me what this statement means, but seems this is referring to feeding stacked random walk matrices through feed-forward neural networks in 4.1.2. I believe this has already been mentioned in GRAPHGPS and GraphiT (i.e. Relative PE using random walks).
- The first step of learning a dense adjacency matrix could be very expensive for large graphs.
- The empirical gain on ZINC and most of the OGB datasets seem quite small compared to the variance.

The evidence to be evaluated is:Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph.

Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Solving it requires far away nodes to communicate, and the communication should consider both the nodes' content (color) and relation within the graph (being in the same row or column).

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

\[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be because RWPE is unable to detect symmetries in the graph. Nodes 0,4,20 and 24 in the example illustrated in Figure 1(a) will all get the same embedding by RWPE.

Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

The average distance between nodes in this dataset is larger than that of any other dataset in our experiments, which can explain why Graph Transformers variations dominate its leaderboards. Since positional encoding was not found to be significant by the previous three highest ranking models in this dataset, and considering the large improvement of our model over GPS, it seems that positional attention brings significant benefits in this benchmark, which allows us to reach a new state-of-the-art.

As we can see in Table 5, adding our positional attention and encoding outperforms both RWSE and LapPE encodings.

Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. \begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Params.**} & \multicolumn{2}{c}{Peptides-func} & Peptides-struct \\ \cline{3-4}  & & **AP \(\uparrow\)** & **MAE \(\downarrow\)** \\ \hline GCN & 508k & 0.5930\(\pm\)0.0023 & 0.3496\(\pm\)0.0013 \\ GINE & 476k & 0.5498\(\pm\)0.0079 & 0.3547\(\pm\)0.0045 \\ GatedGCN & 509k & 0.5864\(\pm\)0.0077 & 0.3420\(\pm\)0.0013 \\ GatedGCN+RWSE & 506k & 0.6069\(\pm\)0.0035 & 0.3357\(\pm\)0.0006 \\ \hline Transformer+LapPE & 488k & 0.6326\(\pm\)0.0126 & 0.2529\(\pm\)0.0016 \\ SAN+LapPE & 493k & 0.6384\(\pm\)0.0121 & 0.2683\(\pm\)0.0043 \\ SAN+RWSE & 500k & 0.6439\(\pm\)0.0075 & 0.2545\(\pm\)0.0012 \\ \hline Graph Diffuser & 509k & **0.6651\(\pm\)0.001** & **0.2461\(\pm\)0.001** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I can assess that the claim "GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation" is mostly well-stated but slightly overstated.

The evidence demonstrates that GD does create Virtual Edges that capture structural relationships and uses them for attention and node representation (shown in Figures 1 and 3). The empirical results in Table 5 support performance improvements. However, per the reviewer comments, some of the gains are small compared to variance, and the mechanism of "learning to combine information propagation" isn't clearly explained. The claim would be more precise if it acknowledged these limitations.

<score>2</score>

Justification:
1. Supporting evidence:
- Figures show clear visualization of how Virtual Edges capture structural information
- Mathematical formulations demonstrate how structural information is incorporated
- Table 5 shows empirical improvements

2. Moderating factors:
- Reviewer notes improvements are "quite small compared to the variance" on some datasets
- Some mechanisms are unclearly explained ("multiple different propagation steps")
- First step could be "very expensive for large graphs"

3. The claim is largely accurate but would benefit from more measured language acknowledging limitations and variance in effectiveness.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation.

The review comment to be evaluated is:

Strength:
1. The proposed method performs favorably on benchmarks datasets.
2. The method is effective but is also quite simple in term of implementing it.
3. The counter example of the 2-d grid histogram counting problem for graph transformer is interesting as all state of the art model fail to solve it.

Weakness:
1. The technical contribution is incremental. Graph diffusion (both in discrete and the ODE form) has been studied and applied in the domain of graph neutral network. While adding graph diffusion to the positional attention is new, it is a trivial extension. [1][2][3]
2. While it is surprising that most graph transformer could not solve the grid based counting problem, this problem itself in the context of graph can be easily solved by diffusion matrix itself if node color is expressed as node feature with one-hot encoding.

[1] https://papers.nips.cc/paper/2019/hash/23c894276a2c5a16470e6a31f4618d73-Abstract.html
[2] https://aclanthology.org/2021.emnlp-main.642.pdf
[3] https://openreview.net/forum?id=EMxu-dzvJk

The evidence to be evaluated is:Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph.

Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Solving it requires far away nodes to communicate, and the communication should consider both the nodes' content (color) and relation within the graph (being in the same row or column).

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

\[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be because RWPE is unable to detect symmetries in the graph. Nodes 0,4,20 and 24 in the example illustrated in Figure 1(a) will all get the same embedding by RWPE.

Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

The average distance between nodes in this dataset is larger than that of any other dataset in our experiments, which can explain why Graph Transformers variations dominate its leaderboards. Since positional encoding was not found to be significant by the previous three highest ranking models in this dataset, and considering the large improvement of our model over GPS, it seems that positional attention brings significant benefits in this benchmark, which allows us to reach a new state-of-the-art.

As we can see in Table 5, adding our positional attention and encoding outperforms both RWSE and LapPE encodings.

Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. \begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Params.**} & \multicolumn{2}{c}{Peptides-func} & Peptides-struct \\ \cline{3-4}  & & **AP \(\uparrow\)** & **MAE \(\downarrow\)** \\ \hline GCN & 508k & 0.5930\(\pm\)0.0023 & 0.3496\(\pm\)0.0013 \\ GINE & 476k & 0.5498\(\pm\)0.0079 & 0.3547\(\pm\)0.0045 \\ GatedGCN & 509k & 0.5864\(\pm\)0.0077 & 0.3420\(\pm\)0.0013 \\ GatedGCN+RWSE & 506k & 0.6069\(\pm\)0.0035 & 0.3357\(\pm\)0.0006 \\ \hline Transformer+LapPE & 488k & 0.6326\(\pm\)0.0126 & 0.2529\(\pm\)0.0016 \\ SAN+LapPE & 493k & 0.6384\(\pm\)0.0121 & 0.2683\(\pm\)0.0043 \\ SAN+RWSE & 500k & 0.6439\(\pm\)0.0075 & 0.2545\(\pm\)0.0012 \\ \hline Graph Diffuser & 509k & **0.6651\(\pm\)0.001** & **0.2461\(\pm\)0.001** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears somewhat overstated. While the evidence shows that GD does use graph diffusion to create "Virtual Edges" that capture structural relationships and influence the Transformer's attention mechanism, the learning aspect is less definitively demonstrated. The evidence mainly describes the mechanism and architecture rather than proving that structural relationships are "learned." The review comments also note that graph diffusion is not novel and the technical contribution is incremental. Additionally, while the method shows good performance, the interpretation of what is "learned" may be overstated.

<score>3</score>

Justification:
1. The evidence shows the mechanism exists but doesn't fully demonstrate "learning" of structural relationships
2. Review comments highlight that graph diffusion techniques are established in prior work
3. While performance results are strong (Table 5), the claim's certainty about what is "learned" goes beyond what's conclusively shown
4. The visualizations and examples (Fig 1, 3) demonstrate the mechanism but don't prove learning of relationships
5. Technical novelty appears incremental rather than fundamentally new learning capabilities


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation.

The review comment to be evaluated is:

Pros:
The experiments can effectively show the performance of the proposed method.

Cons:
1. There are existing works [1, 2] about incorporating graph diffusion in attention-based GNN that are not covered in this work. Besides, in the Related Work section, the subsection "Diffusion" might be revised to "Diffusion in Graph" to avoid any ambiguity with "Diffusion Models".
2. The improvement on many benchmarks is limited compared to the standard deviations.
3. The models in Table 1 are not well described in the main context. For example, the settings of "GNN →Transformer" and "GNN | Transformer+RWPE" can be briefly mentioned in 5.1.1. Moreover, the necessity of designing "GNN →Transformer" and "GNN | Transformer+RWPE" for comparison is not clear.
4. About the figure drawing: Figure 1 is a little hard to understand since it contains terms like "Edge-Wise FFN/projection" and unclear attention maps that are only described in the latter sections. The Transformer architecture in Figure 3 is not well drawn. The Positional Attention module is even partly outside the Multi-Head Attention module.

[1] Wang, Guangtao, Rex Ying, Jing Huang, and Jure Leskovec. "Multi-hop attention graph neural network." arXiv preprint arXiv:2009.14332 (2020).

[2] Liu, Yonghao, Renchu Guan, Fausto Giunchiglia, Yanchun Liang, and Xiaoyue Feng. "Deep attention diffusion graph neural networks for text classification." In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8142-8152. 2021.

The evidence to be evaluated is:Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph.

Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Solving it requires far away nodes to communicate, and the communication should consider both the nodes' content (color) and relation within the graph (being in the same row or column).

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

\[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be because RWPE is unable to detect symmetries in the graph. Nodes 0,4,20 and 24 in the example illustrated in Figure 1(a) will all get the same embedding by RWPE.

Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

The average distance between nodes in this dataset is larger than that of any other dataset in our experiments, which can explain why Graph Transformers variations dominate its leaderboards. Since positional encoding was not found to be significant by the previous three highest ranking models in this dataset, and considering the large improvement of our model over GPS, it seems that positional attention brings significant benefits in this benchmark, which allows us to reach a new state-of-the-art.

As we can see in Table 5, adding our positional attention and encoding outperforms both RWSE and LapPE encodings.

Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. \begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Params.**} & \multicolumn{2}{c}{Peptides-func} & Peptides-struct \\ \cline{3-4}  & & **AP \(\uparrow\)** & **MAE \(\downarrow\)** \\ \hline GCN & 508k & 0.5930\(\pm\)0.0023 & 0.3496\(\pm\)0.0013 \\ GINE & 476k & 0.5498\(\pm\)0.0079 & 0.3547\(\pm\)0.0045 \\ GatedGCN & 509k & 0.5864\(\pm\)0.0077 & 0.3420\(\pm\)0.0013 \\ GatedGCN+RWSE & 506k & 0.6069\(\pm\)0.0035 & 0.3357\(\pm\)0.0006 \\ \hline Transformer+LapPE & 488k & 0.6326\(\pm\)0.0126 & 0.2529\(\pm\)0.0016 \\ SAN+LapPE & 493k & 0.6384\(\pm\)0.0121 & 0.2683\(\pm\)0.0043 \\ SAN+RWSE & 500k & 0.6439\(\pm\)0.0075 & 0.2545\(\pm\)0.0012 \\ \hline Graph Diffuser & 509k & **0.6651\(\pm\)0.001** & **0.2461\(\pm\)0.001** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation."

Review:
The evidence strongly supports this claim through multiple components:
1. The paper demonstrates how Virtual Edges capture structural information between distant nodes
2. Figure 1 clearly illustrates how information from multiple propagation steps is combined
3. The empirical results in Table 5 show improved performance compared to other positional encoding methods
4. The mathematical formulation (equations 4-9) provides concrete mechanisms for how structural relationships are extracted and used
5. The reviewer comments don't contradict this specific claim

While there are some limitations noted in the reviews regarding experimental comparisons, the core claim about the model's ability to extract and use structural relationships is well-supported by both theoretical and empirical evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation.

The review comment to be evaluated is:

Strength:
1. The paper presents two novel design for a graph transformer, the graph diffuser and context attention.
2. The paper provides many experiments in different datasets to demonstrate the performance of their approach.
3. The detail description of the experimental setup is also useful in understanding the benchmarks.

Weakness:
1. The authors should add more literature review. By googling "graph transformer", there is a survey [1] on arXiv that mentioned various methods regarding graph transformer. But some of those methods are not reviewed by the authors. Also, the experiments should include more baselines.
2. When use weighted adjacency, the model does not leverage any information from original adjacency.
3. More ablation study are needed to reveal the importance of each component. For example, context attention only, context attention + weighted adj, context attention + diffuser + weighted adj, and so on.

[1] Min, E., Chen, R., Bian, Y., Xu, T., Zhao, K., Huang, W., ... & Rong, Y. (2022). Transformer for Graphs: An Overview from Architecture Perspective. arXiv preprint arXiv:2202.08455. (https://arxiv.org/abs/2202.08455)

The evidence to be evaluated is:Early Graph Transformer works Dwivedi and Bresson (2020) used the graph Laplacian eigenvectors as the node positional encoding (PE) to provide a sense of nodes' location in the input graph.

Since then, there have been many recent works constructing PE from the Laplacian Lim et al. (2022); Beaini et al. (2021); Wang et al. (2022); Lim et al. (2022), as well as other structural or positional information such as the node degree Ying et al. (2021) and random-walk SE Dwivedi et al. (2022). Relative positioning was applied with great success on a large molecular benchmark by Graphormer Ying et al. (2021); Shi et al. (2022) by using pair-wise graph distances to direct the Transformer's attention. Further, GraphiT Mialon et al. (2021) used relative positions between nodes derived from the random walk kernel to bias the attention but heuristically chose a single random walk length.

Diffusion and other distance measures were used before to increase the expressivity of MP-GNNs Li et al. (2020), replace the local message passing scheme( Atwood and Towsley (2016); Wu et al. (2019); Klicpera et al. (2019, 2018)) or modulate the Transformers attention Mialon et al. (2021).

To the best of our knowledge, this work is the first Graph Transformer to: (1) learn to construct a new adjacency matrix using node and edge features to generate positional or relative encoding, and (2) learn to combine information propagation over multiple different propagation steps in an end-to-end manner.

Solving it requires far away nodes to communicate, and the communication should consider both the nodes' content (color) and relation within the graph (being in the same row or column).

First, GD embeds the structural relations between distant nodes in what we refer to as Virtual Edges. Then, the Transformer processes the nodes while using the virtual edges to direct the computation at different layers. _Virtual Edges_ are high-dimensional representations constructed between distant nodes in the graph. They contain rich information on the structural and positional relationships. To consider relations between distant nodes, the first step is broadening the receptive field on which the architecture operates.

Xu et al. (2018), however, considering multiple powers of the matrix at once, we zoom out of the local Message Passing paradigm and discover information that GNNs may not detect. Virtual Edges contain structural information such as if nodes are on an odd length cycle, can distinguish many non-isomorphic graphs, and cover many distance measures such as shortest distance and generalized PageRank Li et al. (2020). In order to mix information between different propagation steps and extract meaningful structural relations, each stack is processed by a fully connected edge-wise feed-forward network.

Rather than using the original adjacency matrix in equation 1, we found it beneficial to learn a new adjacency matrix using the node and edge features. \[\hat{A_{ij}}=ReLU(BN([x_{i};x_{j},e_{ij}]W_{1}))W_{2} \tag{4}\] \[A_{ij}=normalize(\sigma(\hat{A_{ij}})) \tag{5}\] Where \(;\) means contacting vectors, \(\sigma\) is the sigmoid function and normalize means \(l1\) row normalization. \(x_{i}\),\(x_{j}\in R^{d}\), \(e_{ij}\in R^{d_{edge}}\), \(W_{1}\in R^{(2d+d_{edge})\times 2d}\) and \(W_{2}\in R^{2d\times 1}\). If there are no edge features, we use only the nodes. This aligns with existing approaches such as GAT (Velickovic et al. (2018); Brody et al. (2022)), which find it beneficial to dynamically learn a new adjacency matrix. Most Graph Transformers use the graph structure to either alter the node's representations or to affect the attention mechanism itself.

In the input layer, Self-Virtual Edges are added to the node representation as positional encoding, and at each attention layer, the Virtual Edges are reduced to an attention matrix that is combined with the standard dot-product attention. At each of the Transformer attention layers, we linearly project each virtual edge \(E_{ij}\) separately to get the positional attention score between \(i\) and \(j\).

\[Att=normalize(exp(\hat{Att}^{Content})\odot\sigma(\hat{Att}^{Position})) \tag{8}\] Where \(\sigma\) is the sigmoid function and \(\odot\) means element-wise multiplication. In multi-head attention with \(h\) heads, the projection in equation 6 is done \(h\) times. equation 8 can be viewed as scaling the content attention coefficients based on the positional attention.

They contain important structural information, such as whether the node is part of an odd length cycle and the degrees of adjacent nodes. We project Self-Virtual-Edges to the node's representation dimensionality and add them as positional encoding. \[X_{i}=X_{i}+Relu(E_{ii}W_{pe}) \tag{9}\]

Adding such features to the node representations allows the Transformer to use structural information in all of its layers, including the feed-forward network.

This is not surprising as the Transformer is oblivious to the graph structure.

This may be because RWPE is unable to detect symmetries in the graph. Nodes 0,4,20 and 24 in the example illustrated in Figure 1(a) will all get the same embedding by RWPE.

Looking at Figure 1(b), we can see the model learns to use the different attention mechanisms in an intuitive way, with content attention detecting nodes in the same color and position attention detecting nodes in the same row or column.

The average distance between nodes in this dataset is larger than that of any other dataset in our experiments, which can explain why Graph Transformers variations dominate its leaderboards. Since positional encoding was not found to be significant by the previous three highest ranking models in this dataset, and considering the large improvement of our model over GPS, it seems that positional attention brings significant benefits in this benchmark, which allows us to reach a new state-of-the-art.

As we can see in Table 5, adding our positional attention and encoding outperforms both RWSE and LapPE encodings.

Using a controlled example, we demonstrated its effectiveness in modeling long-range interactions while providing better interpretability. We then showed that this translates to real-world problems by evaluating GD on eight benchmarks from multiple domains. \begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**\# Params.**} & \multicolumn{2}{c}{Peptides-func} & Peptides-struct \\ \cline{3-4}  & & **AP \(\uparrow\)** & **MAE \(\downarrow\)** \\ \hline GCN & 508k & 0.5930\(\pm\)0.0023 & 0.3496\(\pm\)0.0013 \\ GINE & 476k & 0.5498\(\pm\)0.0079 & 0.3547\(\pm\)0.0045 \\ GatedGCN & 509k & 0.5864\(\pm\)0.0077 & 0.3420\(\pm\)0.0013 \\ GatedGCN+RWSE & 506k & 0.6069\(\pm\)0.0035 & 0.3357\(\pm\)0.0006 \\ \hline Transformer+LapPE & 488k & 0.6326\(\pm\)0.0126 & 0.2529\(\pm\)0.0016 \\ SAN+LapPE & 493k & 0.6384\(\pm\)0.0121 & 0.2683\(\pm\)0.0043 \\ SAN+RWSE & 500k & 0.6439\(\pm\)0.0075 & 0.2545\(\pm\)0.0012 \\ \hline Graph Diffuser & 509k & **0.6651\(\pm\)0.001** & **0.2461\(\pm\)0.001** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 1: Illustration of our positional attention, focusing on node B. Information propagation from multiple propagation steps is combined to create Virtual Edges(colored) between distant nodes, which then direct the Transformer’s attention in each layer.

Input: Table 5: Evaluation on the recently suggested Peptides-func and Peptides-struct Dwivedi et al. (2022b).

Input: Figure 3: An illustration of Graph Diffuser with 2 Transformer layers. Virtual Edges are created by combining multiple powers of the weighted adjacency matrix(left). Virtual Edges modulate each of the Transformers attention layers(right). Self-Virtual Edges are added to the Transformer input as positional encoding(bottom right)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim is modestly overstated. While the evidence demonstrates that GD does learn structural relationships and uses them to influence attention, some key points suggest overstatement:

1. The paper shows GD can learn and use structural relationships, but doesn't definitively prove it "extracts" them in the way claimed
2. The evidence about directing Transformer attention is primarily shown through one visualization example (Fig 1b)
3. The reviewer points out that more ablation studies are needed to reveal the importance of components
4. While performance improvements are shown, the direct causal link between structural learning and performance isn't fully established

The core mechanism is supported, but the certainty and comprehensiveness implied by the claim goes beyond what's strictly demonstrated.

<score>3</score>