1. Although the work is important and detailed, from the novelty perspective, it is an extension of norm-based and rollout aggregation methods to another set of residual connections and norm layer in the encoder block. Not a strong weakness, as the work makes a detailed qualitative and quantitative analysis, roles of each component, which is a novelty in its own right.
2. The impact of the work would be more strengthened with the proposed approach's (local and global) applicability to tasks other than classification like question answering, textual similarity, etc. ( Like in the previous work, Kobayashi et al. (2020))
1. For equations 12 and 13, authors assume equal contribution from the residual connection and multi-head attention. However, in previous work by Kobayashi et al. (2021), it is observed and revealed that residual connections have a huge impact compared to mixing (attention). This assumption seems to be the opposite of the observations made previously. What exactly is the reason for that, for simplicity (like assumptions made by Abnar and Zuidema (2020))?
2. At the beginning of the paper, including the abstract and list of contributions, the claim about the components involved is slightly inconsistent with the rest. For instance, the 8th line in abstract is "incorporates all components", line 73 also says the "whole encoder", but on further reading, FFN (Feed forward layers) is omitted from the framework. This needs to be framed (rephrased) better in the beginning to provide a clearer picture.
3. While FFNs are omitted because a linear decomposition cannot be obtained (as mentioned in the paper), is there existing work that offers a way around (an approximation, for instance) to compute the contribution? If not, maybe a line or two should be added that there exists no solution for this, and it is an open (hard) problem. It improves the readability and gives a clearer overall picture to the reader.
4. Will the code be made publicly available with an inference script? It's better to state it in the submission, as it helps in making an accurate judgement that the code will be useful for further research.
3. While FFNs are omitted because a linear decomposition cannot be obtained (as mentioned in the paper), is there existing work that offers a way around (an approximation, for instance) to compute the contribution? If not, maybe a line or two should be added that there exists no solution for this, and it is an open (hard) problem. It improves the readability and gives a clearer overall picture to the reader.