The main concern is that the improvements on GLUE are moderate, considering that the baselines are relatively lower than others. The original BERT uses a batch size of 256 and is trained for 1M steps. Your model uses a batch size of 2K and is trained for 300K steps. It can be seen that your models are trained longer than the original BERT, while your baselines are still lower. Also, it has been widely proven that the original BERT results are quite underestimated. Taking all these into consideration, I cannot say the results of Dict-BERT is significantly better than vanilla BERT.
Training and fine-tuning Dict-BERT is relatively complicated (than vanilla BERT). Considering it outcomes, I am not convinced that Dict-BERT can be generally used and adapted.
As we also need to append the definitions of rare words in fine-tuning stage, it introduces additional computing cost and might not be friendly to long-sequence tasks (such as document classification and machine reading comprehension). Unfortunately, the authors did not discuss these aspects.
Major comments:
Figure 1: I am not sure if the rare words are always masked. At least in this example, Covid-19 is masked. Maybe a clear illustration should be given on whether the rare words are always masked.
Section 3.4.2: You mentioned that the negative sample is RANDOMLY chosen from the whole vocabulary. However, in Figure 1, the negative sample SARS seems to be a very good negative to Covid-19. It has been widely proven that hard negative samples are much useful in discriminative training. Did you try to find a better way rather than randomly choosing a negative sample?
If there are n
rare words, does that mean we have to append n
negative samples accordingly?
As mentioned in weaknesses, Dict-BERT may not be friendly to long-sequence tasks. Have you tried your model on those tasks? (such as SQuAD)
Minor comments:
Section 4.1: we use BERT -> We use BERT
Section 4.2: 252581 -> 252,581
Please try to reorganize the positions of the figures and tables. For example, you first mention Table 1 on page 8, but the actual table 1 is on page 6, which is not friendly to the readers. I have to read back and forth in these sections.
Section 4.6: In Knowledge Attention v.s. Full Attention.. What is Appendix x.x?
252581 -> 252,581 Please try to reorganize the positions of the figures and tables. For example, you first mention Table 1 on page 8, but the actual table 1 is on page 6, which is not friendly to the readers. I have to read back and forth in these sections. Section 4.6: In Knowledge Attention v.s. Full Attention.. What is Appendix x.x?