1. The setting of Unsupervised Online Adaptation is a little bit strange. As described in Sec 3.1, the model requires a training set, including documents, quires and labels. It seems that the adaptation process is NOT "Unsupervised" because the training set also requires annatations.
2. The problem that this paper focuses on may be unrealistic. Figure 8 shows that adapting to test documents leads to performance degradation on unrelated queries. In practice, we expect to update the knowledge of Large Language Models without affecting the performance on general tasks. Besides, existing large scale QA systems, e.g. GPT-4, show strong In-Context Learning abilities. In other words, the model can reason about new documents and answer questions that have never been seen before.
1. The setting of Unsupervised Online Adaptation is a little bit strange. As described in Sec 3.1, the model requires a training set, including documents, quires and labels. It seems that the adaptation process is NOT "Unsupervised" because the training set also requires annatations.