Our observations that the fine-tuning reduces [CLS] attention to the [SEP] token can be interpreted in these terms.
further analysis is needed to understand whether this reduction in attention is specifically due to the task of predicting fMRI recordings or generally arises during fine-tuning on any task.