- Using original encoders as baselines might not be sufficient. In most experiments, the paper only compares with the original XLM-R or mBERT trained without any knowledge base information. It is unclear whether such encoders being fine-tuned towards the KB tasks would actually perform comparable to the proposed approach. I would like to see experiments like just fine tuning the encoders with the same dataset but the MLM objective in their original pretraining and comparing with them. Such baselines can leverage on input sequences as simple as `<s>X_s X_p X_o </s>` where one of them is masked w.r.t. MLM training.
- The design of input formats is intuitive and lacks justifications. Although the input formats for monolingual and cross-lingual links are designed to be consistent, it is hard to tell why the design would be chosen. As the major contribution of the paper, justifying the design choice matters. In other words, it would be better to see some comparisons over some variants, say something like `<s>[S]X_s[S][P]X_p[P][O]X_o[O]</s>` as wrapping tokens in the input sequence has been widely used in the community.
- The abstract part is lengthy so some background and comparisons with prior work can be elaborated in the introduction and related work. Otherwise, they shift perspective of the abstract, making it hard for the audience to catch the main novelties and contributions.
- In line 122, triples denoted as $(e_1, r, e_2)$ would clearly show its tuple-like structure instead of sets.
- In sec 3.2, the authors argue that the Prix-LM (All) model consistently outperforms the single model, hence the ability of leveraging multilingual information. Given the training data sizes differ a lot, I would like to see an ablation that the model is trained on a mix of multilingual data with the same overall dataset size as the monolingual. Otherwise, it is hard to justify whether the performance gain is from the large dataset or from the multilingual training.
- The design of input formats is intuitive and lacks justifications. Although the input formats for monolingual and cross-lingual links are designed to be consistent, it is hard to tell why the design would be chosen. As the major contribution of the paper, justifying the design choice matters. In other words, it would be better to see some comparisons over some variants, say something like `<s>[S]X_s[S][P]X_p[P][O]X_o[O]</s>` as wrapping tokens in the input sequence has been widely used in the community.