1. **The motivation of this paper appears to be questionable.** The authors claim that DoRA increases the risk of overfitting, basing this on two pieces of evidence:
- DoRA introduces additional parameters compared to LoRA.
- The gap between training and test accuracy curves for DoRA is larger than that of BiDoRA.
However, these two points do not convincingly support the claim. First, while additional parameters can sometimes contribute to overfitting, they are not a sufficient condition for it. In fact, DoRA adds only a negligible number of parameters (0.01% of the model size, as reported by the authors) beyond LoRA. Moreover, prior work [1] suggests that LoRA learns less than full fine-tuning and may even act as a form of regularization, implying that the risk of overfitting is generally low across these PEFT methods.
Additionally, the training curves are not necessarily indicative of overfitting, as they can be significantly influenced by factors such as hyperparameters, model architecture, and dataset characteristics. The authors present results from only a single configuration, which limits the generalizability of their findings.
Finally, the authors’ attribution of an *alleged overfitting problem* to DoRA’s concurrent training lacks a strong foundation.
2. **The proposed BiDoRA method is overly complex and difficult to use.** It requires a two-phase training process, with the first phase itself consisting of two sub-steps. It also introduces two additional hyperparameters: the weight of orthogonality regularization and a ratio for splitting training and validation sets. As a result, BiDoRA takes 3.92 times longer to train than LoRA.
3. **Performance differences between methods are minimal across evaluations**. In nearly all results, the performance differences between the methods are less than 1 percentage point, which may be attributable to random variation. Furthermore, the benchmarks selected are outdated and likely saturated.
[1] [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673)
3. **Performance differences between methods are minimal across evaluations**. In nearly all results, the performance differences between the methods are less than 1 percentage point, which may be attributable to random variation. Furthermore, the benchmarks selected are outdated and likely saturated. [1] [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673)