1. I thought the novelty is questionable. The authors claimed that the proposed Uni-Mol is the first pure 3D molecular pretraining framework. However, there have been already a few similar works. For example,
a. The Graph Multi-View Pre-training (GraphMVP) framework leverages the correspondence and consistency between 2D topological structures and 3D geometric views.
Liu et al., Pre-training Molecular Graph Representation with 3D Geometry, ICLR 2021.
b. The geometry-enhanced molecular representation learning method (GEM) proposes includes several dedicated geometry-level self-supervised learning strategies to learn molecular geometry knowledge.
Fang et al., Geometry-enhanced molecular representation learning for property prediction, nature machine intelligence, 2022.
c. Guo et al. proposed a self-supervised pre-training model for learning structure embeddings from protein 3D structures.
Guo et al., Self-Supervised Pre-training for Protein Embeddings Using Tertiary Structures, AAAI 2022.
d. The GeomEtry-Aware Relational Graph Neural Network (GearNet) framework uses type prediction, distance prediction and angle prediction of masked parts for pretaining.
Zhang et al., Protein Representation Learning by Geometric Structure Pretraining, ICML 2022 workshop.
2. The comparison with the SOTA methods may be unfair. The performance of the paper is based on the newly collected 209M dataset. However, the existing methods use smaller datasets. For example, GEM employs only 20M unlabeled data. Because the scale of datasets has a significant impact on the accuracy, the superior of the proposed method may be from the new large-scale datasets.
3. The authors claimed one of the contributions is that the proposed Uni-Mol contains a simple and efficient SE(3)-equivariant Transformer backbone. However, I thought this contribution is too weak.
4. The improvement is not very impressive or convincing. Although with a larger dataset for pretraining, the improvement is a bit limited, e.g., in Table 1.
5. It is not clear which part causes the main improvement: Transformer, pretraining or the larger dataset?
6. It could be better to show the 3D position recovery and masked atom prediction accuracy and visualize the results.
7. The visualization of the self-attention map and pair distance map in Appendix H is interesting. However, according to the visualization, the self-attention map is very similar to the pair distance map, as the author explained. In this case, why not directly use pair distance as attention? Or what does self-attention actually learn besides distance in the task? As self-attention is computationally expensive, is it really needed?
2. The comparison with the SOTA methods may be unfair. The performance of the paper is based on the newly collected 209M dataset. However, the existing methods use smaller datasets. For example, GEM employs only 20M unlabeled data. Because the scale of datasets has a significant impact on the accuracy, the superior of the proposed method may be from the new large-scale datasets.