- The PLMs used (BERT) are by current standards, quite old, and quite small. As work in scaling PLMs up to sizes orders of magnitude greater, performance on syntactic tasks has shown to improve naturally (along with many other useful emergent forms of knowledge). Some comparison to larger models / application of this method to such models, is necessary to ensure that the method has any practical purpose.
- There are also no baselines from existing work. There are other forms of fine-tuning, such as adapters, which seek to add additional knowledge to PLMs with less chance of catastrophic forgetting. The authors even cite one of these papers. This is also a confusing oversight, because the many appropriate inline citations which contrast various decisions in this work to decisions in existing work demonstrate a great familiarity with the literation, so lacking any comparison to any existing methods is an odd oversight.
- Some questionable design choices. Perplexity is used as a measure of the model retaining semantic information after fine-tuning, and while that does relate to the original task, there are also aspects of domain drift which are possible and separate from catastrophic forgetting. How are such factors controlled?
- Related, there is questionable motivation. Often when talking about catastrophic forgetting, both the original training and the new task are both relevant. This is clear in the context of robotics, where learning a new behavior should not result in hindering the robot from performing existing behaviors. Is this true in this case? PPL is almost always not a valuable end goal, and the entire PLM/LLM paradigm is built around this notion of pre-training in whatever way leads to learning useful linguistic representations, before fine-tuning, aligning, or few-shotting the model towards the task the user actually cares about. If users never care about both tasks in approximately equal measure, than what good is retaining the original model weights which were not pertinent to the target task?
- There's arguably too much going on here. The focus of the paper aims to be about catastrophic forgetting, but secondary to that, is also the problem of matching the right syntactic fine-tuning task to the right problem. This is not entirely known a priori, so all possible pairings are explored, but realistically a good guess can be made (as it would likely be if pursued in a more practical setting). For instance, it is no surprise that the phrase syntax task helps with key phrase identification. The disadvantage of the exhaustive approach is that it has both distracted from the main takeaway points while cutting into the space available for supporting the main hypothesis.
- No inclusion of baselines from existing work / SOTA on performance tables
- Key extraction F1 results are better than standard optimizers, but negligibly so.
- No discussion of GC vs EWC. When a priori would you choose which method? If the paper included only one such method, traditional optimizers would be the best choice in most situations.
- Some questionable design choices. Perplexity is used as a measure of the model retaining semantic information after fine-tuning, and while that does relate to the original task, there are also aspects of domain drift which are possible and separate from catastrophic forgetting. How are such factors controlled?