- While studying instability of LIME, the work likely confuses that instability with various
other sources of instability involved in the methods:
- Instability in the model being explained.
- Instability of ranking metrics used.
- Instability of the LIME implementation used. This one drops entire words instead of perturbing
embeddings which would be expected to be more stable. Dropping words is a discrete and
impactful process compared to embedding perturbation.
Suggestion: control for other sources of instability. That is, measure and compare model
instability vs. resulting LIME instability; measure and compare metric instability vs. resulting
LIME instability. Lastly consider evaluating the more continuous version of input perturbation
for LIME based on embeddings. While the official implementation does not use embeddings, it
shouldn't be too hard to adjust it given token embedding inputs.
- Sample complexity of the learning LIME uses to produce explanations is not discussed. LIME
attempts to fit a linear model onto a set of inputs of a model which is likely not linear. Even
if the target model was linear, LIME would need as many samples as there are input features to be
able to reconstruct the linear model == explanation. Add to this likelihood of the target not
being linear, the number of samples needed to estimate some stable approximation increases
greatly. None of the sampling rates discussed in the paper is suggestive of even getting close to
the number of samples needed for NLP models.
Suggestion: Investigate and discuss sample complexity for the type of linear models LIME uses as
there may be even tight bounds on how many samples are needed to achieve close to optimal/stable solution.
Suggestion: The limitations discusses the computational effort is a bottleneck in using larger
sample sizes. I thus suggest investigating smaller models. It is not clear that using
"state-of-the-art" models is necessary to make the points the paper is attempting to make.
- Discussions around focus on changing or not changing the top feature are inconsistent throughout
the work and the ultimate reason behind them is hard to discern. Requiring that the top feature
does not change seems like a strange requirement. Users might not even look at the features below
the top one so attacking them might be irrelevant in terms of confusing user understanding.
"Moreover, its experiment settings are not ideal as it allows perturbations of top-ranked
predictive features, which naturally change the resulting explanations"
Isn't changing the explanations the whole point in testing explanation robustness? You also cite
this point later in the paper:
"Moreover, this requirement also accounts the fact that end-users often consider only the
top k most important and not all of the features"
Use of the ABS metric which focuses on the top-k only also is suggestive of the importance of top
features. If top features are important, isn't the very top is the most important of all? Later:
"... changing the most important features will likely result in a violation to constraint in
Eq. (2)"
Possibly but that is what makes the perturbation/attack problem challenging. The text that
follows does not make sense to me:
"Moreover, this will not provide any meaningful insights to analysis on stability
in that we want to measure how many changes in the perturbed explanation that
correspond to small (and not large) alterations to the document.
I do not follow. The top feature might change without big changes to the document.
Suggestion: Coalesce the discussion regarding the top features into one place and present a
self-consistent argument of where and why they are allowed to be changed or not.
Smaller things:
- The requirement of black-box access should not dismiss comparisons with white-box attacks as baselines.
- You write:
"As a sanity check, we also constrain the final perturbed document to result in at least one
of the top k features decreasing in rank."
This does not sound like a sanity check but rather a requirement of your method. If it were
sanity check, you'd measure whether at least one of the top k features decreased without imposing
it as a requirement.
- The example of Table 5 seems to actually change the meaning significantly. Why was such a change
allowed given "think" (verb) and "thinking" (most likely adjective) changed part of speech?
- You write:
"Evidently, replacing any of the procedure steps of XAIFOOLER with a random mechanism dropped
its performance"
I'm unsure that "better than random" is a strong demonstration of capability.
- You write: "Evidently, replacing any of the procedure steps of XAIFOOLER with a random mechanism dropped its performance" I'm unsure that "better than random" is a strong demonstration of capability.