1) Reducing the vocabulary size is one way of reducing the size of embedding, however, there are other alternatives such as dimensionality reduction (Raunak et al. 2019), quantization (see some works here (Gholami et al. 2021)), bloom embedding (Serra & Karatzoglou 2017), distillation networks (Hinton et al. 2015), etc.
This work should be compared against some of these related baselines to show its true potential as an innovative approach for embedding compactness.
*Raunak, V., Gupta, V., & Metze, F. (2019, August). Effective dimensionality reduction for word embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) (pp. 235-243).
*Serrà, J., & Karatzoglou, A. (2017, August). Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks. In Proceedings of the Eleventh ACM Conference on Recommender Systems (pp. 279-287).
*Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021). A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630.
*Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
2) The perplexity experiments are carried out on obsolete language models (n-gram HMM, RNN) that are rarely used nowadays. To better align the paper with current NLP trends, I believe the authors should showcase their approach using transformer-based (masked) language models.
3) The reliance of this approach on a secondary step (vowel-retrieval) to make the text human-readable again could limit its applicability. It would be interesting to see how this representation would perform on generation tasks such as translation or summarization. Since the vowel-retrieval process is not loss-less (word-error-rate 9 for consonant-only and ~3 for masked-vowel representations), it may cause a drastic drop in the performance of the models on such tasks.
4) In addition, this extra vowel-retrieval step would add to the required computational steps and may actually increase the computational requirements (as opposed to the paper’s claim on saving on computational resources).
2) The perplexity experiments are carried out on obsolete language models (n-gram HMM, RNN) that are rarely used nowadays. To better align the paper with current NLP trends, I believe the authors should showcase their approach using transformer-based (masked) language models.