- The paper does not discuss much about linguistic aspect of the dataset. While their procedures are thoroughly described, analyses are quite limited in that they do not reveal much about linguistic challenges in the dataset as compared to, for example, information extraction. The benefit of pretraining on the target domain seems to be the only consistent finding in their paper and I believe this argument holds in majority of datasets.
- Relating to the first point, authors should describe more about the traits of the experts and justify why annotation must be carried out by the experts, outside its commercial values. Were the experts linguistic experts or domain experts? Was annotation any different from what non-experts would do? Did it introduce any linguistic challenges?
- The thorough description of the processes is definitely a strength, it might be easier to follow if the authors moved some of the details to appendix.
L23: I was not able to find in the paper that single-task models consistently outperformed multi-task models. Could you elaborate a bit more about this?
Table 1: It would be easier if you explain about "Type of Skills" in the caption. It might also be worth denoting number of sentences for your dataset, as Jia et al (2018) looks larger than SkillSpan at a first glance.
Section 3: This section can be improved to better explain which of "skill", "knowledge" and "attitude" correspond to "hard" and "soft" knowledge. Soft skills are referred as attitudes (L180) but this work seems to only consider "skill" and "knowledge" (L181-183 and Figure 1)? This contradicts with the claim that SkillSpan incorporates both hard and soft knowledge.
L403: I don't think it is the field's norm to call it multi-task learning when it is merely solving two sequential labeling problems at a same time.
L527: Is there any justification as to why you suspected that domain-adaptive pre-raining lead to longer spans?
L543: What is continuous pretraining?
L543: "Pre-training" and "pretraining" are not spelled consistently.
- Relating to the first point, authors should describe more about the traits of the experts and justify why annotation must be carried out by the experts, outside its commercial values. Were the experts linguistic experts or domain experts? Was annotation any different from what non-experts would do? Did it introduce any linguistic challenges?