- The paper uses much analysis to justify that the information axis is a good tool to be applied. As pointed out in conclusion, I'm curious to see some related experiments that this information axis tool can help with.
- For Figure 1, I have another angle for explaining why randomly-generated n-grams are far away from the extant words: the characterBERT would explicitly maximize the probability of seen character sequence (implicitly minimize the probability of unseen character sequence). So I guess the randomly generated n-grams would have distant different PPL value with the extant words. This is justified in Section 5.4.
- It would be better to define some notations and give a clear definition of the "information axis", "word concreteness" and also "Markov chain information content".
- Other than UMAP, there are some other tools for analyzing the geometry of high-dimensional representations. I believe the idea is not highly integrated with UMAP. So it would be better to show demonstrate results with other tools like T-SNE.
- Other than UMAP, there are some other tools for analyzing the geometry of high-dimensional representations. I believe the idea is not highly integrated with UMAP. So it would be better to show demonstrate results with other tools like T-SNE.