- If I understand correctly, In Tables 1 and 2, the authors report the best results on the **dev set** with the hyper-parameter search and model selection on **dev set**, which is not enough to be convincing. I strongly suggest that the paper should present the **average** results on the **test set** with clearly defined error bars under different random seeds. - Another concern is that the method may not be practical. In fine-tuning, THE-X firstly drops the pooler of the pre-trained model and replaces softmax and GeLU, then conducts standard fine-tuning. For the fine-tuned model, they add LayerNorm approximation and d distill knowledge from original LN layers. Next, they drop the original LN and convert the model into fully HE-supported ops. The pipeline is too complicated and the knowledge distillation may not be easy to control. - Only evaluating the approach on BERTtiny is also not convincing although I understand that there are other existing papers that may do the same thing. For example, a BiLSTM-CRF could yield a 91.03 F1-score and a BERT-base could achieve 92.8. Although computation efficiency and energy-saving are important, it is necessary to comprehensively evaluate the proposed approach.
- The LayerNorm approximation seems to have a non-negligible impact on the performances for several tasks. I think it is an important issue that is worth exploring.
- I am willing to see other reviews of this paper and the response of the authors. - Line #069: it possible -> it is possible?
- If I understand correctly, In Tables 1 and 2, the authors report the best results on the **dev set** with the hyper-parameter search and model selection on **dev set**, which is not enough to be convincing. I strongly suggest that the paper should present the **average** results on the **test set** with clearly defined error bars under different random seeds.