1. Based on some efforts to reproduce the results on my end, it is not clear how strongly the proposed observation holds which might limit the significance of the contributions of the paper.
Comments and questions: 1. What would constitute distributional generalization in the setting of regression? If I consider the setting of regression for a moment, the phenomenon appears to be less surprising: a reasonably smooth regression model which interpolates the train data would necessarily exhibit distributional generalization in that setting. 2. It would help to see some error bars on the plots in Figure 2B. I tried replicating the toy example (classify CIFAR-10 classes as objects vs animals with label noise on cats) and observed that when label noise is 30% on the train data only 2-5% of cats in test set were being labeled as objects. The network used was a ResNet50 trained to train accuracy 96.2% using SGD with learning rate = 0.1, momentum = 0.9 and weight decay of 5e-4 and trained for ~160 epochs. However, I did observe that when the label noise is increased to 70%, the distributional generalization effect was seen more strongly (test cats labeled objects 60-80% of the time). 3. It would make for a stronger case if the paper reports the numbers observed when the label noise experiment is performed on image-net with 1000 classes as well (at least on the non-tail classes). This would further stress test the conjecture. Even if the phenomenon significantly weakens in this setting, the numbers are worth seeing.
Minor comments: 1. AlexNet top-1 accuracy on ImageNet reported as 56.5%. Isn’t this 63.3%? 2. Another minor comment is on the name used to describe the phenomenon: distributional generalization sounds a bit strong to capture the empirical phenomenon presented. It represents the ideal of the total variation between the test and train distributions of the network’s outputs vanishing to zero which might not be the case. It is hard to draw this conclusion from a few test functions on which the outputs match.
2. Another minor comment is on the name used to describe the phenomenon: distributional generalization sounds a bit strong to capture the empirical phenomenon presented. It represents the ideal of the total variation between the test and train distributions of the network’s outputs vanishing to zero which might not be the case. It is hard to draw this conclusion from a few test functions on which the outputs match.