Weakness: 1. The introduction of the motivation (the concept of in-context bias) is not easy to understand at the very beginning. The paper said: “the pretrained NLM can model much stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples.” Acutally it seems quite natural for me and I did not realize it is a problem until I saw more explanations in section 1.1. 2. The theory is a bit complicated and not easy to follow. 3. The experiments are limited. The authors only conduct the evaluation on sentence similarity tasks and open domain QA tasks. However, there are many other tasks that involve sentence pairs. For example, sentence inference tasks such as MNLI and RTE are common tasks in NLP field. The authors should conduct experiments on more types of sentence pair tasks.
3. The experiments are limited. The authors only conduct the evaluation on sentence similarity tasks and open domain QA tasks. However, there are many other tasks that involve sentence pairs. For example, sentence inference tasks such as MNLI and RTE are common tasks in NLP field. The authors should conduct experiments on more types of sentence pair tasks.