--- I think human studies in interpretability research are mis-represented at L59. * These approaches don't just ask people whether they think an approach is trustworthy. They also ask humans to do things with explanations and that seems to have a better connection to whether or not an explanation really explains model behavior. This follows the version of interpretability from [1]. This paper laments a lack of theoretical foundation to interpretability approaches (e.g., at L241,L275-277) and it acknowledges at multiple points that we don't know what ground truth for feature importance estimates should look like. Doesn't a person have to interpret an explanation at some point a model for it to be called interpretable? It seems like human studies may offer a way to philosophically ground interpretability, but this part of the paper mis-represents that research direction in contrast with its treatment of the rest of the related work. Minor evaluation problems: * Given that there are already multiple samples for all these experiments, what is the variance? How significant are the differences between rankings? I only see this as a minor problem because the differences on the right of figure 4 are quite large and those are what matter most. * I understand why more baseline estimators weren't included: it's expensive. It would be interesting to incorporate lower frequency visualizations like Grad-CAM. These can sometimes give significantly different performance (e.g., as in [3]). I expect it may have significant impact here because a more coarse explanation (e.g., 14x14 heatmap) may help avoid noise that comes from the non-smooth, high frequency, per-pixel importance of the explanations investigated. This seems further confirmed by the visualizations in figure 1 which remove whole objects as pointed out at L264. The smoothness of coarse visualization method seems like it should do something similar, so it would further confirm the hypothesis about whole objects implied at L264. * It would be nice to summarize ROAR into one number. It would probably have much more impact that way. One way to do so would be to look at the area under the test accuracy curves of figure 4. Doing so would obscure richer insights that ROAR would provide, but this is a tradeoff made by any aggregate statistic. Presentation: * L106: This seems to carelessly resolve a debate that the paper was previously careful to leave open (L29). Why can't it be that the distribution has changed? Do any experiments disentangle changes in distribution from removal of information? Things I didn't understand: * L29: I didn't get this till later in the paper. I think I do now, but my understanding might change again after the rebuttal. More detail here would be useful. * L85: Wouldn't L1 regularization be applied to the weights? Is that feature selection? What steps were actually taken in the experiments used in this paper? Did the ResNet50 used have L1 regularization? * L122: What makes this a bit unclear is that I don't know what is and what is not a random variable. Normally I would expect some of these (epsilon, eta) to be constants. Suggestions --- * It would be nice to know a bit more about how ROAR is implemented. Were the new datasets dynamically generated? Were they pre-processed and stored? * Say you start re-training from the same point. Train two identical networks with different random seeds. How similar are the importance estimates from these networks (e.g. using rank correlation similarity)? How similar are the sets of the final 10% of important pixels identified by ROAR across different random seeds? If they're not similar then whatever importance estimator isn't even consistent with itself in some sense. This could be thought of as an additional sanity check and it might help understand why the baseline estimators considered don't do well. [1]: Doshi-Velez, F., & Kim, B. (2017). A Roadmap for a Rigorous Science of Interpretability. ArXiv, abs/1702.08608. Final Evaluation --- Quality: The experiments were thorough and appropriately supported the conclusions. The paper really only evaluate importance estimators using ROAR. It doesn't really evaluate ROAR itself. I think this is appropriate given the strong motivation the paper has and the lack of concensus about what methods like ROAR should be doing. Clarity: The paper could be clearer in multiple places, but it ultimately gets the point across. Originality: The idea is similar to [30] as cited. ROAR uses a similar principle with re-training and this makes it new enough. Significance: This evaluation could become popular, inspire future metrics, and inspire better importance estimators. Overall, this makes a solid contribution. Post-rebuttal Update --- After reading the author feedback, reading the other reviews, and participating in a somewhat in-depth discussion I think we reached some agreement, though not everyone agreed about everything. In particular, I agree with R4's two recommendations for the final version. These changes would address burning questions about ROAR. I still think the existing contribution is a pretty good contribution to NeurIPS (7 is a good rating), though I'm not quite as enthusiastic as before. I disagree somewhat with R4's stated main concern, that ROAR does not distinguish enough between saliency methods. While it would be nice to have more analysis about the differences between these methods, ROAR is only one way to analyze these explanations and one analysis needn't be responsible for identifying differences between all the approaches it analyzes.
* L106: This seems to carelessly resolve a debate that the paper was previously careful to leave open (L29). Why can't it be that the distribution has changed? Do any experiments disentangle changes in distribution from removal of information? Things I didn't understand: