1. Such strategy requires extra parallel data, which might not exist in many datasets/tasks especially during the pre-training stage. The authors did not consider such cases to propose some cheap ways to acquire such parallel data. Also, utilizing the parallel data for training increase the size of context window, which require larger context window for models and might be expensive.
2. The question answering requires the template mapping to transform the question into a masked statement, which might cause the poor generalization to questions that are not 'Wh-types'/transformable.
2. The question answering requires the template mapping to transform the question into a masked statement, which might cause the poor generalization to questions that are not 'Wh-types'/transformable.