I have several concerns regarding this paper.
• Novelty. The authors propose to use Ricci flow to compute the distance between nodes so that to sample edges with respect to that distance. Using Ricci flow for distance computation is a well-studied area (as indicated in related work). The only novel part is that each layer gets a new graph; however, this choice is not motivated (why not to train all layers of GNN on different graphs instead?) and has problems (see next).
• Approach. Computing optimal transport distance is generally an expensive procedure. While authors indicated that it takes seconds to compute it on 36 cores machine, it’s not clear how scalable this method is. I would like to see whether it scales on normal machines with a couple of cores. Moreover, how do you compute exactly optimal transport, because the Sinkhorn method gives you a doubly stochastic matrix (how do you go from it to optimal transport?).
• Algorithm. This is the most obscure part of the paper. First, it’s not indicated how many layers do you use in experiments. This is a major part of your algorithm because you claim that if an edge appears in several layers it means that it’s not adversarial (or that it does not harm your algorithm). In most of the baselines, there are at most 2-3 layers. There are theoretical limitations why GNN with many layers may not work in practice (see, the literature on “GNN oversmoothing”). Considering that you didn’t provide the code (can you provide an anonymized version of the code?) and that your baselines (GCN, GAT, etc.) have similar (or the same) performance as in the original papers (where the number of layers is 2-3), I deduce that your model Ricci-GNN also has this number of layers. With that said, I doubt that it’s possible to make any conclusive results about whether an edge is adversarial or not with 2-3 graphs. Moreover, I would expect to see an experiment on how your approach varies depending on the number of layers. This is a crucial part of your algorithm and not seeing discussion of it in the paper, raises concerns about the validity of experiments.
• Design choices. Another potential problem of your algorithm is that the sampled graphs can become dense. There are hyperparameters \sigma and \beta that control the probabilities and also you limit the sampling only for 2-hop neighborhoods (“To keep graph sparsity, we only sample edges between pairs that are within k hops of each other in G (we always take k = 2 in the experiments).” This is arbitrary and the effect of it on the performance is not clear. How did you select parameters \sigma and \beta? Why k=2? How do you ensure that the sampled graphs are similar to the original one? Does it matter that sampled graphs should have similar statistics to the original graph? I guess, this crucially affects the performance of your algorithm, so I would like to see more experiments on this.
• Datasets. Since this paper is mostly experimental, I would like to see a comparison of this model on more datasets (5-7 in total). Verifying on realistic but small datasets such as Cora and Citeseer limits our intuition about performance. For example, Cora is a single graph of 2.7K nodes. As indicated in [1], “Although small datasets are useful as sanity checks for new ideas, they can become a liability in the long run as new GNN models will be designed to overfit the small test sets instead of searching for more generalizable architectures.” There are many sources of real graphs, you can consider OGB [2] or [3].
• Weak baselines. Another major concern of the validity of the experiments is the choice of the baselines. Neither of GNN baselines (GCN, GAT, etc.) was designed for the defense of adversarial attacks, so choosing them for comparison is not fair. A comparison with previous works (indicated in “Adversarial attack on graphs.” in related work section) is necessary. Moreover, an experiment where you randomly sample edges (instead of using Ricci distance) is desirable to compare the performance against random sampling.
• Ablation. Since you use GCN, why the performance of Ricci-GCN is so different from GCN when there 0 perturbations? For Citeseer the absolute difference is 2% which is quite high for the same models. Also, an experiment with different choices of GNN is desirable.
• Training. Since experiments play important role in this paper, it’s important to give a fair setup for the models in comparison. You write “For each training procedure, we run 100 epochs and use the model trained at 100-th epoch.”. This can disadvantageous for many models. A better way would be to run each model setup until convergence on the training set, selecting the epoch using the validation set. Otherwise, your baselines could suffer from either underfitting or overfitting.
[1] https://arxiv.org/pdf/2003.00982.pdf [2] https://ogb.stanford.edu/ [3] https://paperswithcode.com/task/node-classification ==========
After reading the authors comments.
I applaud the authors for greatly improving their paper via the revision. Now the number of layers is specified and the explanation of having many sampled graphs during training is added, which was missing in the original text and was preventing a full understanding of the reasons why the proposed approach works. Overall, I am leaning toward increasing the score.
I still have several concerns about the practicality of Ricci-GNN. In simple words, the proposed approach uses some metric S (Ricci flow) that dictates how to sample graphs for training. The motivation for using Ricci flow is “that Ricci flow is a global process that tries to uncover the underlying metric space supported by the graph topology and thus embraces redundancy”. This claim cites previous papers, which in turn do not discuss what exactly is meant by “a global process that tries to uncover the underlying metric space”. Spectral embeddings also can be considered as a global metric, so some analysis on what properties of Ricci flow makes it more robust to attacks would be appreciated. Also including random sampling in comparison would confirm that the effect is coming not from the fact that you use more graphs during the training, but from how you sample those graphs. In addition, as the paper is empirical and relies on the properties of Ricci flow which was discussed in previous works and was not addressed in the context of adversarial attacks, having more datasets (especially larger ones) in the experiments would improve the paper.
• Approach. Computing optimal transport distance is generally an expensive procedure. While authors indicated that it takes seconds to compute it on 36 cores machine, it’s not clear how scalable this method is. I would like to see whether it scales on normal machines with a couple of cores. Moreover, how do you compute exactly optimal transport, because the Sinkhorn method gives you a doubly stochastic matrix (how do you go from it to optimal transport?).