While there exist many papers discussing the softmax bottleneck or the stolen probability problem, similar to what the authors found, I personally have not found enough evidence in my work that the problem is really severe.
After all, there are intrinsic uncertainties in the empirical distributions of the training data, and it is quite natural for us to use a smaller hidden dimension size than the vocabulary size, because after all, we call them "word embeddings" for a reason.
I guess what I mean to say here is that the problem is of limited interest to me (which says nothing about a broader audience) because the results agree very well with my expectations.
This is definitely not against the authors because they did a good job in showing this via their algorithm and the concrete experiments.
I feel like the authors could mention and expand on the implications when beam search is used.
Because in reality, especially that many MT models are considered in the paper, greedy search is seldomly used.
In other words, "even if greedy search is used, SPP is not a big deal, let alone that in reality we use beam search", something like that.
Compared to the main text, I am personally more interested in the point brought up at L595.
What implications are there for the training of our models?
How does the gradient search algorithm decide on where to put the word vectors and hidden state vector?
Is there anything we, i.e. the trainers of the NNs, can do to make it easier for the NNs?
Small issues: - L006, as later written in the main text, "thousands" is not accurate here. Maybe add "on the subword level"?
- L010, be predicted - L034, personally, I think "expressiveness" is more commonly used, this happens elsewhere in the paper as well.
- L082, autoencoder - L104, greater than or equal to
- L006, as later written in the main text, "thousands" is not accurate here. Maybe add "on the subword level"?