- [Originality] The winner-take-all property has been widely used in previous works such as NN-based clustering algorithms [1] and it’s unclear how this paper contributes novelly to the understanding of this behavior with its extremely simplified settings, especially since most of the findings have been reported in previous works (Sec 5).
- [Quality] The quality of the paper is unacceptable due to the following issues:
1) The experimental setup is highly insufficient for a top-tier conference like ICLR, with overly simplified network, datasets and analyses (only scalar plots from single neurons instead of e.g. cluster analysis and/or visualization), leaving the results highly inconclusive.
2) The observed winner-take-most (divide-and-conquer) behavior could be simply due to insufficient training as it contradicts the neural collapse theory [2] which predicts the opposite, the collapse of all intraclass clusters, leaving the main results highly questionable.
3) Claiming that training noises are required for generalization on the synthetic dataset (Sec 3.1) is quite problematic, since sufficient training should generally lead to the max-margin solution (which generalizes) even without training noises [3]. The extremely large learning rate (1.0) could be causing the problem.
- [Significance] Given the critical issues in the paper’s originality and quality as stated above, it’s unfortunately hard to conclude that this work is significant or sufficiently promising.
[1] Clustering: A neural network approach, Neural networks, 2010.\
[2] Prevalence of neural collapse during the terminal phase of deep learning training, PNAS, 2020.\
[3] The Implicit Bias of Gradient Descent on Separable Data, JMLR, 2018.
- [Originality] The winner-take-all property has been widely used in previous works such as NN-based clustering algorithms [1] and it’s unclear how this paper contributes novelly to the understanding of this behavior with its extremely simplified settings, especially since most of the findings have been reported in previous works (Sec 5).