+ Nice idea Consistent improvements over cross entropy for hierarchical class structures Improvements w.r.t other competitors (though not consistent) Good ablation study
The improvements are small The novelty is not very significant
More comments:
Figure 1: - It is not clear what distortion is at this stage - It is not clear what perturbed MNist is, and respectively: why is the error of a 3-layer CNN so high (12-16% error are reported)? CNNs with 2-3 layers can solve MNist with accuracy higher than 99.5%? - This figure cannot be presented on page 2 without proper definitions. It should be either presented on page 5, where the experiment is defined, or better explained
Page 4: It is said that s can be computed efficiently and this is shown in the appendix, but the version I have do not have an appendix Page 6: the XE+EMD method is not present in a comprehensible manner. 1) p_k symbols are used without definition (tough I think I these are the network predictions p(\hat{y}=k I) 2) the relation of the formula presented to the known EMD is not clear. The latter is a problem solved as linear programming or similar, and not a closed form formula 3) it is not clear what the role of \mu is and why can be set to 3 irrespective of the scale of metric D page 7: The experiments show small, but consistent improvements of the suggested method over standard cross entropy, and improvements versus most competitors in most cases
I have read the reviews of others and the author's response. My main impression of the work remains as it was: that it is nice idea with small but significant empirical success. However, my acquaintance with the previous literature in this subject is partial compared to the acquaintance of other reviewers, so It may well be possible that they are in a better position than me to see the incremental nature of the proposed work. I therefore reduce the rating a bit, to become closer to the consensus.
1) p_k symbols are used without definition (tough I think I these are the network predictions p(\hat{y}=k I) 2) the relation of the formula presented to the known EMD is not clear. The latter is a problem solved as linear programming or similar, and not a closed form formula