1. It seems to me that a very straightforward hypothesis about these two parts would be that the trivial part is what’s very simple, either highly consistent to what’s in the training set, or the images with very typical object pose in the center of the images; and for the impossible part, it might be the images with ambiguous labels, atypical object pose or position. I think the human test results would support this hypothesis, but I wonder whether the authors could provide more evidence to either prove or disprove this hypothesis. 2. The figure 6 is very confusing to me. The caption says that the right part is original ImageNet test set, but the texts on the image actually say it’s the left part. If the texts on the image are right, then the right panel is the consistency on the validation images between the two parts. If I understand the experiments correctly, these results are for models trained on ImageNet training set without the trivial or the impossible part and then tested on ImageNet validation set without the two parts. Although it’s good to see the lower consistency, it should be compared to the consistency between models trained on the whole ImageNet training set and tested on ImageNet validation set without the two parts, which I cannot find. Is the consistency lower because of the changed training process or the changed validation set? 3. It is also unclear how surprising we should be towards the consistency distribution, is this a result of an exponential distribution of the general “identification” difficulty (most images are simple, then less and less are more difficult)?
1. It seems to me that a very straightforward hypothesis about these two parts would be that the trivial part is what’s very simple, either highly consistent to what’s in the training set, or the images with very typical object pose in the center of the images; and for the impossible part, it might be the images with ambiguous labels, atypical object pose or position. I think the human test results would support this hypothesis, but I wonder whether the authors could provide more evidence to either prove or disprove this hypothesis.