1. it utilizes quantization technology but does not compare with the other quantization approaches.
2. there is still a softmax computation left in the presented approach.
3. energy consumption is only estimated within the attention module, even though the reduction in a Transformer block is added (17%), the reduction of the full model (with the classifier) is not reported. I have concerns that the overall reduction may not be sufficiently significant.
4. L1 norm between vectors of 2 matrices needs a high memory cost than the matrix multiplication.
5. experiments were performed on 3 language pairs with the base setting, I wonder whether the approach can perform well in challenging settings (Transformer Big and deep Transformers).
6. the maximum performance loss (- 0.78 BLEU) might be a significant loss.
It's recommended to report the overall energy reduction of the full model, conduct experiments on more language pairs and with the big setting on a few of them.
1. it utilizes quantization technology but does not compare with the other quantization approaches.