1. **Triviality of Convergence Proof**: The theoretical proof for convergence appears trivial. Although the paper claims that $Z$ is non-i.i.d., Assumption 4.1 indicates that $X$ is i.i.d., leading to a clear covariance matrix for $Z$ as $A^\top A / np$. Following Modification 1 in Appendix C, previous theorems can be trivially adapted with straightforward modifications. Thus, the convergence proof lacks substantial novelty and rigor.
2. **Limited Parameterization**: According to Equation 2.2, it seems that only the final logits layer contains parameters $\beta$, while the preceding $S^M X$ lacks parameters. The absence of parameters in these earlier layers raises concerns about why only the last layer is parameterized, which could lead to over-smoothing due to unparameterized iterations of $S X$ and consequently limit the model’s expressiveness.
3. **Basic Transfer Learning Approach**: The transfer learning method employed, a simple $\delta$ fine-tuning, appears overly basic. There is little exploration of alternative, established methods in transfer learning or meta-learning that could potentially enhance the model’s adaptability and robustness.
4. **Issues in Hyperparameter Sensitivity Testing**: The sensitivity experiments on hyperparameters are limited. For instance, in the $\lambda$ experiment, the model fails to achieve the optimal solution seen at $M=5$. Additionally, the range of $\lambda$ tested is narrow; a broader, exponential scale (e.g., 0.01, 0.001, 0.0001) would provide a more comprehensive understanding of the model’s sensitivity.
5. **Lack of Notational Clarity**: The notation lacks clarity and could benefit from a dedicated section outlining all definitions. Many symbols, such as $X_j$, are undefined in Appendix A. A coherent notation guide would improve readability and help readers follow the technical details more effectively.
1. **Triviality of Convergence Proof**: The theoretical proof for convergence appears trivial. Although the paper claims that $Z$ is non-i.i.d., Assumption 4.1 indicates that $X$ is i.i.d., leading to a clear covariance matrix for $Z$ as $A^\top A / np$. Following Modification 1 in Appendix C, previous theorems can be trivially adapted with straightforward modifications. Thus, the convergence proof lacks substantial novelty and rigor.