1. Generalizability: both fine-tuning and in-context learning strategies seem to be tailored for shifting model attention to a smaller chunk of key information, which is confirmed by the attention weight analysis. This makes me to worry to what extent this method could be generalized to other datasets where the information is not presented in contrastive pairs, and where the conflicting information is not restricted to one or two sentences but widely spread in the entire passage. For example, the task of identifying conflicting sentences might have over-simplified the reasoning task by taking the short-cut to ignore lots information, which just happens to be trivial in these specific datasets.
2. The developed strategies, while interesting, might just be marginally relevant to the cognitive process of heuristic / analytical dual passes of human reasoning. The heuristic reasoning process is more related to the information being utilized and the amount of attention paid to more fine-grained details. For instance, in online language comprehension, comprehenders might ignore fine-grained syntactic structured and rely on the semantic meaning of the words and their prior knowledge to interpret "the hearty meal was devouring..." as "the hearty meal was devoured..." (Kim and Osterhout, 2005). However, the heuristic process is less concerned with gratuity of final decision, as implied by the HAR model. The HAR framework breaks the reasoning tasks into multiple sub-tasks, where the gratuity of the decision gradually becomes finer-grained. This might be better characteristics as step-by-step chain of reasoning rather than heuristic decision-making.
3. The ICL-HAR, while improving consistency and verifiability, has greatly impedes the accuracy scores (dropping from 70.4 to 55.6 on TRIP). This should be discussed or at least acknowledged in the main text in more detail.
3. The ICL-HAR, while improving consistency and verifiability, has greatly impedes the accuracy scores (dropping from 70.4 to 55.6 on TRIP). This should be discussed or at least acknowledged in the main text in more detail.