Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# SoTTA: Robust Test-Time Adaptation on Noisy Data Streams

Taesik Gong\({}^{\dagger}\)1  Yewon Kim\({}^{\ddagger}\)1  Taeckyung Lee\({}^{\ddagger*}\)  Sorn Chottananurak\({}^{\ddagger}\)  Sung-Ju Lee\({}^{\ddagger}\)

\({}^{\dagger}\)Nokia Bell Labs \({}^{\ddagger}\)KAIST

taesik.gong@nokia-bell-labs.com

{yewon.e.kim,taeckyung,sorn111930,profsj}@kaist.ac.kr

Equal contribution.

Footnote 1: footnotemark:

###### Abstract

Test-time adaptation (TTA) aims to address distributional shifts between training and testing data using only unlabeled test data streams for continual model adaptation. However, most TTA methods assume benign test streams, while test samples could be unexpectedly diverse in the wild. For instance, an unseen object or noise could appear in autonomous driving. This leads to a new threat to existing TTA algorithms; we found that prior TTA algorithms suffer from those noisy test samples as they blindly adapt to incoming samples. To address this problem, we present Screening-out Test-Time Adaptation (SoTTA), a novel TTA algorithm that is robust to noisy samples. The key enabler of SoTTA is two-fold: (i) input-wise robustness via high-confidence uniform-class sampling that effectively filters out the impact of noisy samples and (ii) parameter-wise robustness via entropy-sharpness minimization that improves the robustness of model parameters against large gradients from noisy samples. Our evaluation with standard TTA benchmarks with various noisy scenarios shows that our method outperforms state-of-the-art TTA methods under the presence of noisy samples and achieves comparable accuracy to those methods without noisy samples. The source code is available at [https://github.com/taeckyung/SoTTA](https://github.com/taeckyung/SoTTA).

## 1 Introduction

Deep learning has achieved remarkable performance in various domains [6; 8; 33], but its effectiveness is often limited when the test and training data distributions are misaligned. This phenomenon, known as domain shift [31], is prevalent in real-world scenarios where unexpected environmental changes and noises result in poor model performance. For instance, in autonomous driving, weather conditions can change rapidly. To address this challenge, Test-Time Adaptation (TTA) [1; 5; 29; 38; 39; 44] has emerged as a promising paradigm that aims to improve the generalization ability of deep learning models by adapting them to test samples, without requiring further data collection or labeling costs.

While TTA has been acknowledged as a promising method for enhancing the robustness of machine learning models against domain shifts, the evaluation of TTA frequently relies on the assumption that the test stream contains only benign test samples of interest. However, test data can be unexpectedly diverse in real-world settings, containing not only relevant data but also extraneous elements that are outside the model's scope, which we refer to _noisy_ samples (Figure 1). For instance, unexpected noises can be introduced in autonomous driving scenarios, such as dust on the camera or adversarial samples by malicious users. As shown in Figure 2, we found that most of the prior TTA algorithms showed significantly degraded accuracy with the presence of noisy samples (e.g., 81.0% \(\rightarrow\) 52.1% in TENT [38] and 82.2% \(\rightarrow\) 54.8% in CoTTA [39]).

To ensure the robustness of TTA against noisy samples, an intuitive solution might be screening out noisy samples from the test stream. Out-of-distribution (OOD) detection [10; 11; 18; 19; 20; 21; 24; 25; 43] is a representative method for this, as it tries to detect whether a sample is drawn from the same distribution as the training data or not. Similarly, open-set domain adaptation (OSDA) [30; 35] and universal domain adaptation (UDA) [34; 42] generalize the adaptation scenario by assuming that unknown classes are present in test data that are not in training data. However, these methods require access to a whole batch of training data and unlabeled target data, which do not often comply with TTA settings where the model has no access to train data at test time due to privacy issues [38] and storing a large batch of data is often infeasible due to resource constraints [12]. Therefore, how to make online TTA robust under practical noisy settings is still an open question.

In this paper, we propose Screening-out Test-Time Adaptation (SoTTA) that is robust to noisy samples. SoTTA achieves robustness to noisy samples in two perspectives: (i) _input-wise_ robustness and (ii) _parameter-wise_ robustness aims to filter out noisy samples so that the model will be trained only with benign samples. We achieve this goal via High-confidence Uniform-class Sampling (HUS) that avoids selecting noisy samples when updating the model (Section 3.1). Parameter-wise robustness pursues updating the model weights in a way that prevents model drifting due to large gradients caused by noisy samples. We achieve this via entropy-sharpness minimization (ESM) that makes the loss landscape smoother and parameters resilient to weight perturbation caused by noisy samples (Section 3.2).

We evaluate SoTTA with three common TTA benchmarks (CIFAR10-C, CIFAR100-C, and ImageNet-C [9]) under four noisy scenarios with different levels of distributional shifts: Near, Far, Attack, and Noise (Section 2). We compare SoTTA with eight state-of-the-art TTA algorithms [1; 17; 27; 28; 29; 38; 39; 44], including the latest studies that address temporal distribution changes in TTA [1; 28; 29; 39; 44]. SoTTA showed its effectiveness with the presence of noisy samples. For instance, in CIFAR10-C, SoTTA achieved 80.0% accuracy under the strongest shift case (Noise), which is a 22.3%p improvement via TTA and 6.4%p better than the best baseline [44]. In addition, SoTTA achieves comparable performance to state-of-the-art TTA algorithms without noisy samples, e.g., showing 82.2% accuracy when the best baseline's accuracy is 82.4% in CIFAR10-C.

**Contributions.** (i) We highlight that test sample diversity in real-world scenarios is an important problem but has not been investigated yet in the literature. We found that most existing TTA algorithms undergo significant performance degradation with sample diversity. (ii) As a solution, we propose SoTTA that is robust to noisy samples by achieving input-wise and parameter-wise robustness. (iii) Our evaluation with three TTA benchmarks (CIFAR10-C, CIFAR100-C, and ImageNet-C) show that SoTTA outperforms the existing baselines.

Figure 1: Unlike prior assumptions (Clean TTA), real-world test streams could include unexpected noisy samples out of the model’s scope (Noisy TTA), such as glare, fallen leaf covering the lens, unseen objects (e.g., a flamingo), and noise in autonomous driving scenarios. The accuracy of existing TTA methods degrades in such cases.

## 2 Preliminaries

**Test-time adaptation.** Let \(\mathcal{D}_{\mathcal{S}}=\{\mathcal{X}^{\mathcal{S}},\mathcal{Y}\}\) be source data and \((\mathbf{x}_{i},y_{i})\in\mathcal{X}^{\mathcal{S}}\times\mathcal{Y}\) be each instance and the label pair that follows a probability distribution of the source data \(P_{\mathcal{S}}(\mathbf{x},y)\). Similarly, let \(\mathcal{D}_{\mathcal{T}}=\{\mathcal{X}^{\mathcal{T}},\mathcal{Y}\}\) be target data and \((\mathbf{x}_{j},y_{j})\in\mathcal{X}^{\mathcal{T}}\times\mathcal{Y}\) be each target instance and the label pair following a target probability distribution \(P_{\mathcal{T}}(\mathbf{x},y)\), where \(y_{j}\) is usually unknown to the learning algorithm. The covariate shift assumption [31] is given between source and target data distributions, which is defined as \(P_{\mathcal{S}}(\mathbf{x})\neq P_{\mathcal{T}}(\mathbf{x})\) and \(P_{\mathcal{S}}(y|\mathbf{x})=P_{\mathcal{T}}(y|\mathbf{x})\). Given an off-the-shelf model \(f(\cdot;\Theta)\) pre-trained from \(\mathcal{D}_{\mathcal{S}}\), the (fully) test-time adaptation (TTA) [38] aims to adapt \(f(\cdot;\Theta)\) to the target distribution \(P_{\mathcal{T}}\) utilizing only \(\mathbf{x}_{j}\) given test time.

**Noisy test samples.** We define _noisy_ test samples to represent any samples that are not included in the target data distribution, i.e., \(\mathbf{\tilde{x}}\notin\mathcal{X}^{\mathcal{T}}\). We use the term noisy to distinguish it from out-of-distribution (OOD), as TTA typically aims to adapt to OOD samples, such as corrupted ones. Theoretically, there could be numerous categories of noisy samples. In this study, we consider five scenarios: Benign, Near, Far2, Attack, and Noise. Figure 3 shows examples of these scenarios. Benign is the typical setting of TTA studies without noisy samples, Near represents a semantic shift [41] from target distribution, Far is a severe shift where covariate shift is evident [41], Attack refers to intelligently generated adversarial attack with perturbation [40], and Noise refers to random noise. We focus on these scenarios to understand the impact of noisy samples in TTA as well as the simplicity of analysis. Detailed settings are described in Section 4.

Footnote 2: We borrowed the term Near and Far from the OOD detection benchmark [41].

## 3 Methodology

**Problem and challenges.** Prior TTA methods assume test samples are benign and blindly adapt to incoming batches of test samples. The presence of noisy samples during test time can significantly degrade their performance for those TTA algorithms, which has not been explored in the literature yet. Dealing with noisy samples in TTA scenarios is particularly challenging as (i) TTA has no access to the source data, (ii) no labels are given for target test data, and (iii) the model is continually adapted and thus a desirable solution should apply to varying models. This makes it difficult to apply existing solutions that deal with a similar problem. For instance, out-of-distribution (OOD) detection studies [11; 19; 20; 43] are built on the assumption that a model is fixed at test time, and open-set domain adaptation (OSDA) methods [30; 35; 42] require labeled source and unlabeled target data for training.

Figure 4: Overview of SoTTA. SoTTA achieves _input-wise robustness_ via high-confidence uniform-class sampling (HUS) and _parameter-wise robustness_ via entropy-sharpness minimization (ESM).

Figure 3: Five test sample scenarios considered in this work: Benign, Near, Far, Attack, and Noise.

**Methodology overview.** To address the problem, we propose Screening-out Test-Time Adaptation (SoTTA), whose overview is described in Figure 4. SoTTA achieves robustness to noisy samples in two perspectives: (i) _input-wise_ robustness via high-confidence uniform-class sampling that avoids selecting noisy samples when updating the model (Section 3.1), and (ii) _parameter-wise_ robustness via entropy-sharpness minimization that makes parameters resilient to weight perturbation caused by noisy samples (Section 3.2).

### Input-wise robustness via high-confidence uniform-class sampling

Our first approach is to ensure input-wise robustness to noisy samples by filtering out them when selecting samples for adaptation. As locating noisy samples without their labels is challenging, our idea is based on the empirical observation of the model predictions with respect to noisy samples.

**Observation.** Our hypothesis is that noisy samples have distinguished properties from benign samples due to the distributional shifts, and this could be observable via the models' prediction outputs. We investigate two types of features that work as proxies for identifying benign samples: (i) confidence of the samples and (ii) predicted class distributions. Specifically, we compared the distribution of the softmax confidence (Figure 4(a)) and the predicted class distribution (Figure 4(b)) of benign samples with noisy samples. First, the confidence of the samples is relatively lower than that of benign samples. The more severe the distribution shift, the lower the confidence (e.g., Far is less confidence than Near), which is also in line with findings of previous studies that pre-trained models show higher confidence on target distribution than out-of-distribution data [10; 21]. Second, we found that noisy samples are often skewed in terms of predictions, and this phenomenon is prominent in more severe shifts (e.g., Noise), except for Attack, whose objective is to make the model fail to correctly classify. These skewed distributions could lead to an undesirable bias in \(p(y)\) and thus might negatively impact the TTA objective, such as entropy minimization [38].

```
0: test data stream \(\mathbf{x}_{t}\), memory \(M\) with capacity \(N\) for test time \(t\in\{1,\cdots,T\}\)do \(\hat{y}_{t}\gets f(\mathbf{x};\Theta)\) if\(C(\mathbf{x};\Theta)>C_{0}\)then\(\triangleright\) Sampling confident data if\(|M|<N\)then  Add \((\mathbf{x}_{t},\hat{y}_{t})\) to \(M\) else \(\mathcal{Y}^{*}\leftarrow\) the most prevalent class(es) in \(M\) if\(\hat{y}_{t}\notin\mathcal{Y}^{*}\)then  Randomly discard \((\mathbf{x}_{i},\hat{y}_{i})\) from \(M\) where \(\hat{y}_{i}\in\mathcal{Y}^{*}\) else  Randomly discard \((\mathbf{x}_{i},\hat{y}_{i})\) from \(M\) where \(\hat{y}_{i}=\hat{y}_{t}\)  Add \((\mathbf{x}_{t},\hat{y}_{t})\) to \(M\)
```

**Algorithm 1** High-confidence Uniform-class Sampling (HUS)

**Solution.** Based on the aforementioned empirical analysis, we propose High-confidence Uniform-class Sampling (HUS) that avoids using noisy samples for adaptation by utilizing a small memory. We maintain confident samples while balancing their predicted classes in the memory. The selected samples in the memory are then used for adaptation. We describe the procedure as a pseudo-code code in Algorithm 1. Given a target test sample \(\mathbf{x}\), HUS measures its confidence. Specifically, we define the confidence \(C(\mathbf{x};\Theta)\) of each test sample \(\mathbf{x}\) as:

\[C(\mathbf{x};\Theta)=\max_{i=1\cdots n}\left(\frac{e^{\hat{y}_{i}}}{\sum_{j=1 }^{n}e^{\hat{y}_{j}}}\right)\text{ where }\hat{y}=f(\mathbf{x};\Theta). \tag{1}\]

Figure 5: Model predictions on benign (CIFAR10-C) and noisy samples.

We store the sample if its confidence is higher than the predefined threshold \(C_{0}\). In this way, we maintain only high-confidence samples in the memory used for adaptation and thus reduce the impact of low-confidence noisy samples.

Furthermore, while storing data in the memory, we balance classes among them. Specifically, if the predicted class of the current test sample is not in the most prevalent class in the memory, then HUS randomly replaces one random sample in the most prevalent class with the new sample. Otherwise, if the current sample belongs to the most prevalent class in the memory, HUS replaces one random sample in the same class with the current one. We can maintain classes uniformly with this strategy, which is effective for not only filtering out noisy samples but also removing class biases among samples when used for adaptation, which we found is beneficial for TTA. We found these two memory management strategies not only effectively reduce the impact of noisy samples for adaptation but also improve the model performance in benign-only cases by avoiding model drifting due to biased and low-confidence samples (Section 4).

With the stored samples in the memory, we update the normalization statistics and affine parameters in batch normalization (BN) [13] layers, following the prior TTA methods [29, 38, 44]. This is known as not only computationally efficient but also showed comparable performance improvement to updating the whole layers. While we aim to avoid using noisy samples for adaptation, a few noisy samples could still be stored in the memory, e.g., when they are similar to benign samples or outliers. To be robust to temporal variances of the samples in the memory, we take the exponential moving average to update the BN statistics (means \(\mu\) and variances \(\sigma^{2}\)) instead of directly using the statistics from the samples in the memory. Specifically, we update the means and variances of BN layers as: (i) \(\hat{\mu}_{t}=(1-m)\hat{\mu}_{t-1}+m\mu_{t}\) and (ii) \(\hat{\sigma}_{t}^{2}=(1-m)\hat{\sigma}_{t-1}^{2}+m\sigma_{t}^{2}\), where \(m\in[0,1]\) is a momentum hyperparameter. We describe updating the affine parameters in Section 3.2.

### Parameter-wise robustness via entropy-sharpness minimization

Our second approach is to secure parameter-wise robustness to noisy samples by training the model in a way that is robust to noisy samples. Our idea is based on the observation that the parameter update is often corrupted with noisy samples, and this could be mitigated by smoothing the loss landscape with respect to parameters.

**Observation.** While most existing TTA algorithms utilize the entropy minimization loss [5, 38, 44], it could drift the model with high gradient samples [29]. We observed that adaptation with noisy samples often hinders the model from adapting to benign samples. Figure 6 shows an example of this phenomenon. Specifically, test samples consist of 10k benign samples and 10k noisy samples (Noise), which are randomly shuffled. We computed the cumulative accuracy for the benign test data and the gradient norm of noisy samples at each step. As shown in Figure 5(a), the gradient norm of noisy samples dropped rapidly, indicating that the model is gradually adapting to these noise data. This leads to a significant accuracy drop for benign samples. The key question here is how to prevent the model from overfitting to noisy samples. Figure 5(b) shows the result with entropy-sharpness minimization (ESM) that we explain in the following paragraph. With ESM, the gradient norm of noise data remains high, and the accuracy for benign samples improved after adaptation as intended.

**Solution.** To make the model parameters robust to adaptation with noisy samples, the entropy loss landscape should be smoother so that the model becomes resilient to unexpected model drift due to noisy samples. To that end, we jointly minimize the naive entropy loss and the sharpness of the entropy loss and thus make the loss landscape robust to model weight perturbations by large gradients from noisy samples. Specifically, we replace the naive entropy minimization \(E\) with the

Figure 6: Cumulative accuracy (%) of benign samples (CIFAR10-C) and gradient norm of noisy samples (Noise) as the adaptation proceeds. The dotted line refers to the source model accuracy.

entropy-sharpness minimization (ESM) as:

\[\min_{\Theta}E_{S}(\mathbf{x},\Theta)=\min_{\Theta}\max_{||t||\leq\rho}E(\mathbf{x };\Theta+\epsilon), \tag{2}\]

where the entropy-sharpness \(E_{S}(\mathbf{x},\Theta)\) is defined as the maximum objective around the weight perturbation with L2-norm constraint \(\rho\). To tackle this joint optimization problem, we follow sharpness-aware minimization [4] similar to [29], which originally aims to improve the generalizability of models over standard optimization algorithms such as stochastic gradient descent (SGD). We repurpose this optimization to make the model robust to noisy samples in TTA scenarios.

Specifically, assuming \(\rho\ll 1\), the optimization can be approximated via a first-order Taylor expansion:

\[\epsilon^{*}(\Theta)\triangleq\operatorname*{arg\,max}_{||\epsilon||2\leq \rho}E(\mathbf{x};\Theta+\epsilon)\approx\operatorname*{arg\,max}_{||\epsilon|| 2\leq\rho}\epsilon^{T}\nabla_{\Theta}E(\mathbf{x};\Theta). \tag{3}\]

The solution for this approximation problem is given by the classical dual norm problem:

\[\hat{\epsilon}(\Theta)=\rho\ \text{sign}(\nabla_{\Theta}E(\mathbf{x};\Theta)) \ |\nabla_{\Theta}E(\mathbf{x};\Theta)|^{q-1}\big{/}\left(||\nabla_{\Theta}E( \mathbf{x};\Theta)||_{q}^{q}\right)^{1/p}, \tag{4}\]

where \(1/p+1/q=1\). We set \(p=2\) for further implementation, following the suggestion [4]. By substituting \(\hat{\epsilon}(\Theta)\) to the original entropy-sharpness minimization problem, the final gradient approximation is:

\[\nabla_{\Theta}E_{S}(\mathbf{x},\Theta)\approx\nabla_{\Theta}E(\mathbf{x}, \Theta)|_{\Theta+\hat{\epsilon}(\Theta)}. \tag{5}\]

In summary, we calculate the entropy-sharpness minimization objective via two steps. First, at time step \(t\), it calculates the \(\hat{\epsilon}(\Theta_{t})\) with previous parameters and entropy loss. It generates the temporal model with the new parameters: \(\Theta_{t}+\hat{\epsilon}(\Theta_{t})\). Second, based on the temporary model, the second step updates the original model's parameters with the approximation in Equation 5. By putting it together with HUS (Section 3.1), parameters are updated by:

\[\Theta_{t}=\Theta_{t-t_{0}}-\eta\nabla_{\Theta}E(\mathbf{x},\Theta)|_{\mathbf{ x}=\text{HUS}_{C_{0}}(t),\ \Theta=\Theta_{t-t_{0}}+\hat{\epsilon}(\Theta_{t-t_{0}})}, \tag{6}\]

where \(\eta\) is the step size and \(t_{0}\) is the model adaptation interval. For simplicity, we set the model adaptation interval the same as the memory capacity, i.e., \(t_{0}=N\). In the experiments, we use \(N=64\), which is one of the most widely-used batch sizes in TTA methods [36; 38; 44].

## 4 Experiments

This section describes our experimental setup and demonstrates the results in various settings. Please refer to Appendix A and B for further details.

**Scenario.** To mimic the presence of noisy samples in addition to the original target test samples, we injected various noisy datasets into target datasets and randomly shuffled them, which we detail in the following paragraphs. We report the classification accuracy of the original target samples to measure the performance of the model in the presence of noisy samples. We ran all experiments with three random seeds (0, 1, 2) and reported the average accuracy and standard deviations in Appendix B.

**Target datasets.** We used three standard TTA benchmarks: **CIFAR10-C**, **CIFAR100-C**, and **ImageNet-C**[9] as our target datasets. All datasets contain 15 different types and five levels of corruption, where we use the most severe corruption level of five. For each corruption type, the CIFAR10-C/CIFAR100-C dataset has 10,000 test data with 10/100 classes, and the ImageNet-C dataset has 50,000 test data with 1000 classes. We use ResNet18 [8] as the backbone network. We pre-trained the model for CIFAR10 with training data and used the TorchVision [23] pre-trained model for ImageNet.

**Noisy datasets.** Besides the target datasets (**Benign**) mentioned above, we consider four noisy scenarios (Figure 3): CIFAR100 [15]/ImageNet [3] (**Near**), MNIST [26] (**Far**), adversarial attack (**Attack**), and uniform random noise (**Noise**). As both CIFAR10-C and ImageNet-C have real-world images for object recognition tasks, CIFAR100 would be a near dataset to them. For CIFAR100-C, we select ImageNet [3] for a near dataset. We select MNIST as a far dataset because they are not real-world images. We referred to the OOD benchmark [41] for choosing the term Near and Far and the datasets. For the adversarial attack, we adopt the Distribution Invading Attack (DIA), which is an adversarial attack algorithm designed for TTA [40]. We inject the small perturbations to malicious samples to increase the overall error rate on benign data in the same batch. For the uniform random noise, we generate pixel-wise uniform-random images with the same size as the target images. We set the number of noisy samples equal to each target dataset as default, and we also investigated the impact of the number of noisy samples in the following ablation study (Figure 7). In cases where the number of noisy samples is different from the target datasets, we randomly resampled them. To ensure that the learning algorithm does not know the information of noisy samples beforehand, the pixel values of all noisy images are normalized with respect to the target dataset.

**Baselines.** We consider various state-of-the-art TTA methods as baselines. **Source** evaluates the pre-trained model directly on the target data without adaptation. Test-time batch normalization (**BN stats**) [27] updates the BN statistics from the test batch. Pseudo-Label (**PL**) [17] optimizes the trainable BN parameters via pseudo labels. Test entropy minimization (**TENT**) [38] updates the BN parameters via entropy minimization.

We also consider the latest TTA algorithms that improve robustness to temporal distribution changes in test streams to understand their performance to noisy samples. Laplacian adjusted maximum-likelihood estimation (**LAME**) [1] modifies the classifier output probability without modifying model internal parameters. Continual test-time adaptation (**CoTTA**) [39] uses weight-averaged and

augmentation-averaged predictions and avoids catastrophic forgetting by stochastically restoring a part of the model. Efficient anti-forgetting test-time adaptation (**EATA**) [28] uses entropy and diversity weight with Fisher regularization to prevent forgetting. Sharpness-aware and reliable optimization (**SAR**) [29] removes high-entropy samples and optimizes entropy with sharpness minimization [4]. Robust test-time adaptation (**RoTTA**) [44] utilizes the teacher-student model to stabilize while selecting the data with category-balanced sampling with timeliness and uncertainty.

**Hyperparameters.** We adopt the hyperparameters of the baselines from the original paper or official codes. We use the test batch size of 64 in all methods for a fair comparison. Accordingly, we set the memory size to 64 and adapted the model for every 64 samples for our method and RoTTA [44]. We conduct TTA in an online manner. We used a fixed hyperparameter of BN momentum \(m=0.2\) and updated the BN affine parameters via the Adam optimizer [14] with a fixed learning rate of \(l=0.001\) and a single adaptation epoch. The confidence threshold \(C_{0}\) is set to 0.99 for CIFAR10-C, 0.66 for CIFAR100-C, and 0.33 for ImageNet-C. We set the sharpness threshold \(\rho=0.05\) as previous works [4, 29]. We specify further details of the hyperparameters in Appendix A.

**Overall result.** Table 1 shows the result on CIFAR10-C for 15 types of corruptions under five noisy scenarios described in Figure 3. We observed significant performance degradation in most of the baselines under the noisy settings. In addition, the extent of accuracy degradation from the Benign case is more prominent in more severe shift cases (e.g., the degree of degradation is generally Near < Far < Attack < Noise). Popular TTA baselines (BN stats, PL, and TENT) might fail due to updating with noisy samples. State-of-the-art TTA baselines that address temporal distribution shifts in TTA (LAME, CoTTA, EATA, SAR, and RoTTA) still suffered from noisy scenarios, as they are not designed to deal with unexpected samples. On the other hand, SoTTA showed its robustness across different scenarios. This validates the effectiveness of our approaches to ensure both input-wise and parameter-wise robustness. Interestingly, we found that SoTTA showed comparable performance to state-of-the-art baselines in the Benign case as well. Our interpretation of this result is two-fold. First, our high-confidence uniform-class sampling strategy filters not only noisy samples but also benign samples that would negatively impact the algorithm's objective. This implies that there exist samples that are more beneficial for adaptation, which aligns with the findings that high-entropy samples

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Method & Benign & Near & Far & Attack & Noise & Avg. \\ \hline Source & \(33.2\pm 0.4\) & \(33.2\pm 0.4\) & \(33.2\pm 0.4\) & \(33.2\pm 0.4\) & \(33.2\pm 0.4\) & \(33.2\pm 0.4\) \\ BN Stats [27] & \(53.7\pm 0.2\) & \(50.8\pm 0.1\) & \(46.8\pm 0.1\) & \(29.2\pm 0.4\) & \(28.3\pm 0.3\) & \(41.8\pm 0.1\) \\ PL [17] & \(56.6\pm 0.2\) & \(48.0\pm 0.3\) & \(42.8\pm 0.7\) & \(39.0\pm 0.4\) & \(23.8\pm 0.6\) & \(42.1\pm 0.3\) \\ TENT [38] & \(59.5\pm 0.0\) & \(46.4\pm 1.4\) & \(40.0\pm 1.3\) & \(31.9\pm 0.7\) & \(20.0\pm 0.9\) & \(39.5\pm 0.7\) \\ LAME [1] & \(31.0\pm 0.5\) & \(31.5\pm 0.5\) & \(30.8\pm 0.7\) & \(31.0\pm 0.6\) & \(31.1\pm 0.7\) & \(31.1\pm 0.6\) \\ CoTTA [39] & \(55.8\pm 0.4\) & \(50.0\pm 0.3\) & \(42.4\pm 0.4\) & \(37.2\pm 0.2\) & \(27.3\pm 0.3\) & \(42.6\pm 0.2\) \\ EATA [28] & \(23.5\pm 1.9\) & \(6.1\pm 0.3\) & \(4.8\pm 0.5\) & \(3.7\pm 0.6\) & \(2.4\pm 0.2\) & \(8.1\pm 0.3\) \\ SAR [29] & \(57.3\pm 0.3\) & \(55.4\pm 0.1\) & \(51.2\pm 0.1\) & \(34.4\pm 0.3\) & \(38.1\pm 1.2\) & \(47.3\pm 0.3\) \\ RoTTA [44] & \(48.7\pm 0.6\) & \(49.4\pm 0.5\) & \(49.8\pm 0.9\) & \(51.5\pm 0.4\) & \(48.3\pm 0.5\) & \(49.6\pm 0.6\) \\ SoTTA & \(\mathbf{60.5\pm 0.0}\) & \(\mathbf{57.1\pm 0.2}\) & \(\mathbf{59.0\pm 0.4}\) & \(\mathbf{61.9\pm 0.8}\) & \(\mathbf{58.6\pm 1.0}\) & \(\mathbf{59.4\pm 0.3}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Classification accuracy (%) on CIFAR100-C. **Bold** numbers are the highest accuracy. Averaged over three different random seeds for 15 types of corruption.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Method & Benign & Near & Far & Attack & Noise & Avg. \\ \hline Source & \(14.6\pm 0.0\) & \(14.6\pm 0.0\) & \(14.6\pm 0.0\) & \(14.6\pm 0.0\) & \(14.6\pm 0.0\) & \(14.6\pm 0.0\) \\ BN Stats [27] & \(27.1\pm 0.0\) & \(18.9\pm 0.1\) & \(14.8\pm 0.0\) & \(17.4\pm 0.8\) & \(12.8\pm 0.0\) & \(18.2\pm 0.1\) \\ PL [17] & \(30.5\pm 0.1\) & \(6.9\pm 0.0\) & \(5.1\pm 0.2\) & \(18.1\pm 1.3\) & \(3.4\pm 0.6\) & \(12.8\pm 0.2\) \\ TENT [38] & \(27.1\pm 0.0\) & \(18.9\pm 0.1\) & \(14.8\pm 0.0\) & \(17.4\pm 0.8\) & \(12.8\pm 0.0\) & \(18.2\pm 0.1\) \\ LAME [1] & \(14.4\pm 0.0\) & \(14.4\pm 0.1\) & \(14.4\pm 0.0\) & \(14.0\pm 0.6\) & \(14.3\pm 0.0\) & \(14.3\pm 0.1\) \\ CoTTA [39] & \(32.2\pm 0.1\) & \(23.3\pm 0.2\) & \(17.6\pm 0.2\) & \(28.3\pm 1.3\) & \(16.0\pm 0.9\) & \(23.4\pm 0.2\) \\ EATA [28] & \(38.0\pm 0.1\) & \(25.6\pm 0.4\) & \(23.1\pm 0.1\) & \(26.1\pm 0.1\) & \(20.7\pm 0.2\) & \(26.7\pm 0.0\) \\ SAR [29] & \(36.1\pm 0.1\) & \(27.6\pm 0.3\) & \(23.5\pm 0.4\) & \(26.8\pm 1.0\) & \(22.0\pm 0.4\) & \(27.2\pm 0.2\) \\ RoTTA [44] & \(29.7\pm 0.0\) & \(25.6\pm 0.4\) & \(29.2\pm 0.2\) & \(32.0\pm 1.2\) & \(31.2\pm 0.2\) & \(29.5\pm 0.3\) \\ SoTTA & \(\mathbf{39.8\pm 0.0}\) & \(\mathbf{27.9\pm 0.3}\) & \(\mathbf{36.1\pm 0.1}\) & \(\mathbf{41.1\pm 0.1}\) & \(\mathbf{39.0\pm 0.1}\) & \(\mathbf{36.8\pm 0.0}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Classification accuracy (%) on ImageNet-C. **Bold** numbers are the highest accuracy. Averaged over three different random seeds for 15 types of corruption.

harm adaptation performance [29]. Second, entropy-sharpness minimization helps ensure both the robustness to noisy samples and the generalizability of the model by preventing model drifts from large gradients, leading to performance improvement with benign samples. We found similar patterns for the CIFAR100-C (Table 2) and ImageNet-C (Table 3). More details are in Appendix B.

**Impact of individual components of SoTTA.** We conducted an ablative study to further investigate the effectiveness of SoTTA's individual components. Table 4 shows the result of the ablation study for CIFAR10-C. For input-wise robustness, **HC** refers to high-confidence sampling, and **UC** refers to uniform-class sampling. The two strategies are integrated into our high-confidence uniform-class sampling (**HUS**). **ESM** is our entropy-sharpness minimization for parameter-wise robustness. Note that we utilized FIFO memory with the same size for the UC case without HC and the native entropy minimization where we did not utilize ESM. Overall, the accuracy is improved as we sequentially added each approach of SoTTA. This validates our claim that ensuring both input-wise and parameter-wise robustness via HUS and ESM is a synergetic strategy to combat noisy samples in TTA.

**Impact of the number of noisy samples.** We also investigate the effect of the size of noisy samples on TTA algorithms. Specifically, for the CIFAR10-C dataset with the Noise case, we varied the noise samples from 5,000 (5k) to 20,000 (20k) while fixing the size of benign samples as 10,000. Figure 7 shows the result. We observe that the accuracy of most baselines tends to deteriorate with a larger number of noisy samples and is sometimes even worse than without adaptation (Source). SoTTA showed its resilience to the size of noisy samples with 1.9%p degradation from 5k to 20k samples. RoTTA also showed its robustness to noisy samples to some extent, but the performance gain is around 6.4%p lower than SoTTA.

## 5 Related work

**Test-time adaptation.** While test-time adaptation (TTA) attempts to optimize model parameters with unlabeled test data streams, it lacks consideration of spoiling of the sample itself; i.e., it inevitably adapts to potential outlier samples mixed in the stream, such as adversarial data or mere noise. Most existing TTA algorithms directly optimize the model with incoming sample data. Test-time normalization [27, 36] updates the batch norm statistics in test-time to minimize the expected loss. TENT [38] minimizes the entropy of the model's predictions on a batch of test data.On the one hand, recent studies promote the robustness of the model, yet the consideration is limited to temporal distribution shifts of test data [1, 4, 5, 28, 29, 39, 44]. For instance, CoTTA [39] tries to adapt the model in a continually changing target environment via self-training and stochastical restoring. NOTE [5] proposes instance-aware batch normalization combined with prediction-balanced reservoir sampling to ensure model robustness towards temporally correlated data stream, which requires retraining with source data. We provide a method-wise comparison with EATA [28], SAR [29], and RoTTA [44] in Appendix D.2. To conclude, while existing TTA methods seek the robustness of the

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Method & Benign & Near & Far & Attack & Noise & Avg. \\ \hline Source & 57.7 ± 1.0 & 57.7 ± 1.0 & 57.7 ± 1.0 & 57.7 ± 1.0 & 57.7 ± 1.0 & 57.7 ± 1.0 & 57.7 ± 1.0 \\ HC & 34.9 ± 4.8 & 13.6 ± 0.3 & 17.6 ± 3.8 & 16.9 ± 1.6 & 16.8 ± 0.2 & 20.0 ± 2.0 \\ UC & 66.4 ± 3.0 & 62.1 ± 0.8 & 56.5 ± 2.0 & 70.0 ± 3.9 & 59.5 ± 3.0 & 62.9 ± 0.7 \\ HC + UC (HUS) & 69.8 ± 1.1 & 61.7 ± 1.3 & 58.4 ± 0.5 & 40.9 ± 5.5 & 58.9 ± 2.6 & 57.9 ± 0.8 \\ ESM & **82.6 ± 0.2** & 77.9 ± 0.4 & 72.8 ± 0.7 & 83.4 ± 0.2 & 60.5 ± 1.8 & 75.4 ± 0.5 \\ HC + ESM & 82.3 ± 0.2 & 80.9 ± 0.6 & 74.9 ± 2.4 & 83.5 ± 0.2 & 68.7 ± 7.0 & 78.0 ± 2.0 \\ UC + ESM & 82.2 ± 0.2 & 78.0 ± 0.4 & 75.9 ± 0.5 & 84.3 ± 0.1 & 77.7 ± 0.7 & 79.6 ± 0.2 \\ HUS + ESM (SoTTA) & 82.2 ± 0.3 & **81.4 ± 0.5** & **81.6 ± 0.6** & **84.5 ± 0.2** & **80.0 ± 1.4** & **81.9 ± 0.5** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Classification accuracy (%) and corresponding standard deviation of varying ablative settings in SoTTA on CIFAR10-C. **Bold** numbers are the highest accuracy. Averaged over three different random seeds for 15 types of corruption.

Figure 7: Classification accuracy (%) varying the size of noisy samples on CIFAR10-C under Noise.

model to temporal distribution shifts, they do not consider scenarios where noisy samples appear in test streams.

**Out-of-distribution detection.** Out-of-distribution (OOD) detection [10; 11; 18; 19; 20; 21; 24; 25; 43] aims to ensure the robustness of the model by identifying when a given data falls outside the training distribution. A representative method is a thresholding approach [10; 19; 20; 21], that defines a scoring function given an input and pretrained classifier. The sample is detected as OOD data if the output of the scoring function is higher than a threshold. Importantly, OOD detection studies are built on the condition that training and test domains are the same, which differs from TIA's scenario. Furthermore, OOD detection assumes that a model is fixed during test time, while a model changes continually in TTA. These collectively make it difficult to apply OOD detection studies directly to TTA scenarios.

**Open-set domain adaptation.** Open-set Domain Adaptation (OSDA) assumes that a target domain contains unknown classes not discovered in a source domain [30; 35] in domain adaptation scenarios. These methods aim to learn a mapping function between the source and target domains. While the target scenario that a model could encounter unknown classes of data in the test time is similar to our objective, these methods do not fit into TTA as it assumes both labeled source and unlabeled target data are available in the training time. Universal domain adaptation (UDA) [34; 42] further generalizes the assumption of OSDA by allowing unknown classes to present in both the source and the target domains. However, the same problem still remains as it requires labeled source and unlabeled target data in training time, which do not often comply with TTA settings where the model has no access to train data at test time due to privacy issues [38].

## 6 Discussion and conclusion

We investigate the problem of having noisy samples and the performance degradations caused by those samples in existing TTA methods. To address this issue, we propose SoTTA that is robust to noisy samples by high-confidence uniform-class sampling and entropy-sharpness minimization. Our evaluation with four noisy scenarios reveals that SoTTA outperforms state-of-the-art TTA methods in those scenarios. We believe that the takeaways from this study are a meaningful stride towards practical advances in overcoming domain shifts in test time.

**Limitations and future work.** While we focus on four noisy test stream scenarios, real-world test streams might have other types of sample diversities that are not considered in this work. Furthermore, recent TTA algorithms consider various temporal distribution shifts, such as temporally-correlated streams [5] and domain changes [39]. Towards developing a TTA algorithm robust to any test streams in the wild, more comprehensive and realistic considerations should be taken into account, which we believe is a meaningful future direction.

**Potential negative societal impacts.** As TTA requires continual computations for every test sample for adaptation, environmental concerns might be raised such as carbon emissions [37]. Recent studies such as memory-economic TTA [12] might be an effective way to mitigate this problem.

## Acknowledgments and Disclosure of Funding

This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00495, On-Device Voice Phishing Call Detection).

## References

* [1] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 8344-8353, June 2022.
* [2] Dhanajit Brahma and Piyush Rai. A probabilistic framework for lifelong test-time adaptation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 3582-3591, June 2023.
* [3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In _2009 IEEE Conference on Computer Vision and Pattern Recognition_, pages 248-255, 2009.
* [4] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In _International Conference on Learning Representations_, 2021.
* [5] Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. NOTE: Robust continual test-time adaptation against temporal correlation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_, 2022.
* [6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. _Commun. ACM_, 63(11):139-144, oct 2020.
* [7] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. Saul, Y. Weiss, and L. Bottou, editors, _Advances in Neural Information Processing Systems_, volume 17. MIT Press, 2004.
* [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, June 2016.
* [9] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In _International Conference on Learning Representations_, 2019.
* [10] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In _International Conference on Learning Representations_, 2017.
* [11] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. _Proceedings of the International Conference on Learning Representations_, 2019.
* [12] Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. MECTA: Memory-economic continual test-time model adaptation. In _The Eleventh International Conference on Learning Representations_, 2023.
* [13] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning_, volume 37 of _Proceedings of Machine Learning Research_, pages 448-456, Lille, France, 07-09 Jul 2015. PMLR.
* [14] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In _International Conference on Learning Representations (ICLR)_, 2015.
* [15] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. _Master's thesis, Department of Computer Science, University of Toronto_, 2009.
* [16] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. _Proceedings of the IEEE_, 86(11):2278-2324, 1998.
* [17] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In _Workshop on challenges in representation learning, ICML_, page 896, 2013.

* [18] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In _International Conference on Learning Representations_, 2018.
* [19] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems_, volume 31. Curran Associates, Inc., 2018.
* [20] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In _International Conference on Learning Representations_, 2018.
* [21] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_, volume 33, pages 21464-21475. Curran Associates, Inc., 2020.
* [22] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In _International Conference on Learning Representations (ICLR)_, 2017.
* [23] TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. [https://github.com/pytorch/vision](https://github.com/pytorch/vision), 2016.
* [24] Yifei Ming, Ying Fan, and Yixuan Li. POEM: Out-of-distribution detection with posterior sampling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 15650-15665. PMLR, 17-23 Jul 2022.
* [25] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(04):5216-5223, Apr. 2020.
* [26] Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. _arXiv preprint arXiv:1906.02337_, 2019.
* [27] Zachary Nado, Shreyas Padhy, D Sculley, Alexander D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. _arXiv preprint arXiv:2006.10963_, 2020.
* [28] Shuaiecheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 16888-16905. PMLR, 17-23 Jul 2022.
* [29] Shuaiecheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In _The Eleventh International Conference on Learning Representations_, 2023.
* [30] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_, Oct 2017.
* [31] Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. _Dataset shift in machine learning_. Mit Press, 2008.
* [32] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, June 2016.
* MICCAI 2015_, pages 234-241, Cham, 2015. Springer International Publishing.
* [34] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_, volume 33, pages 16282-16292. Curran Associates, Inc., 2020.
* [35] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In _Proceedings of the European Conference on Computer Vision (ECCV)_, September 2018.
* [36] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_, volume 33, pages 11539-11551. Curran Associates, Inc., 2020.
* [37] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. _Commun. ACM_, 63(12):54-63, nov 2020.
* [38] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In _International Conference on Learning Representations_, 2021.
* [39] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 7201-7211, June 2022.
* [40] Tong Wu, Feiran Jia, Xiangyu Qi, Jiachen T. Wang, Vikash Sehwag, Saeed Mahloujifar, and Prateek Mittal. Uncovering adversarial risks of test-time adaptation. _arXiv preprint arXiv:2301.12576_, 2023.
* [41] Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, WENXUAN PENG, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_, volume 35, pages 32598-32611. Curran Associates, Inc., 2022.
* [42] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Universal domain adaptation. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2715-2724, 2019.
* [43] Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximum classifier discrepancy. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_, October 2019.
* [44] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 15922-15932, June 2023.

Experiment details

We conducted all experiments in the paper using three random seeds (0, 1, 2) and reported the average accuracies and their corresponding standard deviations. The experiments were performed on NVIDIA GeForce RTX 3090 and NVIDIA TITAN RTX GPUs. For a single execution of SoTTA, the test-time adaptation phase consumed 1 minutes for CIFAR10-C/CIFAR100-C and 10 minutes for ImageNet-C.

### Baseline details

In this study, we utilized the official implementations of the baseline methods. To ensure consistency, we adopted the reported best hyperparameters documented in the respective papers or source code repositories. Furthermore, we present supplementary information regarding the implementation specifics of the baseline methods and provide a comprehensive overview of our experimental setup, including detailed descriptions of the employed hyperparameters.

SoTTA (Ours).We used ADAM optimizer [14], with a BN momentum of \(m=0.2\), and learning rate of \(l=0.001\) with a single adaptation epoch. We set the HUS size to 64 and the confidence threshold \(C_{0}\) to 0.99 for CIFAR10-C (10 classes), 0.66 for CIFAR100-C (100 classes), and 0.33 for ImageNet-C (1,000 classes). We set entropy-sharpness L2-norm constraint \(\rho=0.5\) following the suggestion [4].

Pl.For PL [17], we only updated the BN layers following the previous studies [38, 39]. We set the learning rate as \(LR=0.001\) as the same as TENT [38].

Tent.For TENT [38], we set the learning rate as \(LR=0.001\) for CIFAR10-C and \(LR=0.00025\) for ImageNet-C, following the guidance provided in the original paper. We referred to the official code3 for implementations.

Footnote 3: [https://github.com/DequanWang/tent](https://github.com/DequanWang/tent)

Lame.LAME [1] relies on an affinity matrix and incorporates hyperparameters associated with it. We followed the hyperparameter selection specified by the authors in their paper and referred to their official code4 for implementation details. Specifically, we employed the kNN affinity matrix with a value of k set to 5.

Footnote 4: [https://github.com/fiveai/LAME](https://github.com/fiveai/LAME)

CoTTA [39] incorporates three hyperparameters: the augmentation confidence threshold \(p_{th}\), restoration factor \(p\), and exponential moving average (EMA) factor \(m\). To ensure consistency, we adopted the hyperparameter values recommended by the authors. Specifically, we set the restoration factor to \(p=0.01\) and the EMA factor to \(\alpha=0.999\). For the augmentation confidence threshold, the authors provide a guideline for its selection, suggesting using the 5% quantile of the softmax predictions' confidence on the source domains. We followed this guideline, which results in \(p_{th}=0.92\) for CIFAR10-C, \(p_{th}=0.72\) for CIFAR100-C, and \(p_{th}=0.1\) for ImageNet-C. We referred to the official code5 for implementing CoTTA.

Footnote 5: [https://github.com/qinenergy/cotta](https://github.com/qinenergy/cotta)

EATA.For EATA [28], we followed the settings from the original paper. We set \(LR=0.005/0.005/0.00025\) for CIFAR10-C/CIFAR100-C/ImageNet-C, entropy constant \(E_{0}=0.4\times\ln\lvert\mathcal{Y}\rvert\) where \(\lvert\mathcal{Y}\rvert\) is number of classes. We set cosine sample similarity threshold \(\epsilon=0.4/0.4/0.05\), trade-off parameter \(\beta=1/1/2,000\), the moving average factor \(\alpha=0.1\). We utilized 2,000 samples for calculating Fisher importance as suggested. We referred to the official code6 for implementing EATA.

Footnote 6: [https://github.com/mr-eggplant/EATA](https://github.com/mr-eggplant/EATA)

Sar.SAR [29] aims to adapt to diverse batch sizes, and we chose a typical batch size of 64 for a fair comparison. We followed the learning rate as \(LR=0.00025\), sharpness threshold \(\rho=0.5\), and entropy threshold \(E_{0}=0.4\times\ln\lvert\mathcal{Y}\rvert\) where \(\lvert\mathcal{Y}\rvert\) is the total number of classes, as suggested in the original paper. Finally, we froze the top layer (layer4 for ResNet18) as the original paper, and SoTTA also follows this implementation. We referred to the original code7 for implementing SAR.

Footnote 7: [https://github.com/mr-eggplant/SAR](https://github.com/mr-eggplant/SAR)
RoTTA.RoTTA [44] uses Adam Optimizer by setting learning rate as \(LR=0.001\) and \(\beta=0.9\). We followed the authors' hyperparameters selection from the paper, including BN-statistic exponential moving average updating rate as \(\alpha=0.05\), the Teacher model's exponential moving average updating rate as \(\nu=0.001\), timeliness parameter as \(\lambda_{t}=1.0\), and uncertainty parameter as \(\lambda_{u}=1.0\). We referred to the original code8 for implementing RoTTA.

Footnote 8: [https://github.com/BIT-DA/RoTTA](https://github.com/BIT-DA/RoTTA)

### Target dataset details

CIFAR10-C/CIFAR100-C.CIFAR10-C/CIFAR100-C [9] serves as a widely adopted benchmark for evaluating the robustness of models against corruptions [27, 36, 38, 39]. Both datasets consist of 50,000 training samples and 10,000 test samples, categorized into 10/100 classes. To assess the robustness of models, datasets introduce 15 types of corruptions to the test data, including Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Frosted Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transformation, Pixelate, and JPEG Compression. For our experiments, we adopt the highest severity level of corruption, level 5, in line with previous studies [27, 36, 38, 39]. Consequently, the datasets consist of 150,000 corrupted test samples. To train our models, we employ the ResNet18 [8] architecture as the backbone network. The model is trained on the clean training data to generate the source models. We utilize stochastic gradient descent with a momentum of 0.9 and cosine annealing learning rate scheduling [22] for 200 epochs. The initial learning rate is set to 0.1, and a batch size 128 is used during training.

ImageNet-C.ImageNet-C is another widely adopted benchmark for evaluating the robustness of models against corruptions [1, 27, 36, 38, 39]. The ImageNet dataset [3] consists of 1,281,167 training samples and 50,000 test samples. Similar to CIFAR10-C, ImageNet-C applies the same 15 types of corruptions, resulting in 750,000 corrupted test samples. We utilize the highest severity level of corruption, equivalent to CIFAR10-C. For our experiments, we employ a pre-trained ResNet18 [8] model from the TorchVision library [23], which is pre-trained on the ImageNet dataset [3] and is widely used as a backbone for various computer vision tasks.

### Noisy dataset details

CIFAR100 (Near).CIFAR100 [15] consists of 50,000/10,000 training/test data with 100 classes. We utilized training data without any corruption. We undersampled the dataset to 10,000 for the CIFAR10-C and CIFAR100-C target cases by randomly removing samples and used the entire training set (50,000) for the ImageNet-C target case.

ImageNet (Near).ImageNet [3] consists of 1,281,167/50,000 training/test data with 1,000 classes. We utilized test data without any corruption. We undersampled the dataset to 10,000 for the CIFAR100-C target case by randomly removing samples.

MNIST (Far).MNIST [16] contains 60,000/10,000 training/test data with 10 classes. We utilized test data without any corruption. We used the entire test set for the CIFAR10-C/CIFAR100-C target case, and oversampled the dataset by randomly resampling, which results in 50,000 samples that is equivalent to the size of each ImageNet-C target data.

Attack.We implemented the modified indiscriminate distribution invading attack (DIA) [40]. First, we duplicated the entire set of target samples and treated them as malicious samples. Subsequently, we randomly shuffled these duplicated samples within the original target sample set. During the adaptation phase, we injected perturbations into the malicious samples to increase the overall error rate on benign samples within the same batch. As a result, we perturbed 10,000 samples (CIFAR10-C/CIFAR100-C) and 50,000 samples (ImageNet-C) to serve as attack samples. For CIFAR10-C/CIFAR100-C, we used hyperparameters of maximum perturbation constraint \(\epsilon=0.1\), attack learning rate \(\alpha=1/255\), and attacking steps \(N=10\). For ImageNet-C, we used hyperparameters of maximum perturbation constraint \(\epsilon=0.2\), attack learning rate \(\alpha=1/255\), and attacking steps \(N=1\).

Uniform random noise (Noise).We generated a uniform random valued image in the scaled RGB range \([0,1]\), with the same height and width as the corresponding target dataset. We generated the same amount of noise samples as each target dataset.

Result details

Additional ablative studies

We conducted experiments to understand the sensitivity of our two hyperparameters: confidence threshold (\(C_{0}\)) and BN momentum (\(m\)). We varied \(C_{0}\) and \(m\) and reported the corresponding accuracy.

Confidence threshold.Our result shows that the selection of \(C_{0}\) shows similar patterns across different scenarios (Benign \(\sim\) Noise). The result illustrates a tradeoff; a low \(C_{0}\) value does not effectively reject noisy samples, while a high \(C_{0}\) value filters benign data. We found a proper value of \(C_{0}\) (0.99) that generally works well across the scenarios. Also, we found that the optimal \(C_{0}\) depends primarily on in-distribution data. Our interpretation is that setting different \(C_{0}\) values for CIFAR10-C, CIFAR100-C, and ImageNet-C is straightforward as they have a different number of classes (10, 100, and 1,000), which leads to different ranges of the model's confidence.

BN momentum.Across the tested range, the variations in performance were found to be negligible. This finding indicates that choosing a low momentum value from within the specified range ([0.05, 0.3]) is adequate to maintain a favorable performance. Please note that setting a high momentum would corrupt the result, which is implicated by the algorithms directly utilizing test-time statistics (e.g., TENT) suffering from accuracy degradation with noisy data streams (e.g., TENT: 81.0% \(\rightarrow\) 52.1% for Noise at Table 1).

## Appendix D Further discussions

### Theoretical explanation of the impact of noisy data streams

We provide a theoretical explanation of the impact of noisy data streams with a common entropy minimization as an example. With the Bayesian-learning-based frameworks [2; 7], we can express the posterior distribution \(p\) of the model in terms of training data \(D\) and benign test data \(B\) in test-time adaptation:

\[\log p(\theta|D,B)=\log q(\theta)-\frac{\lambda_{B}}{|B|}\sum_{b=1}^{|B|}H(y_{ b}|x_{b}). \tag{7}\]

The posterior distribution of model parameters depends on the prior distribution \(q\) and the average of entropy \(H\) of benign samples with a multiplier \(\lambda\). Here, we incorporate the additional noisy data stream \(N\) into Equation 7 and introduce a new posterior distribution considering noisy streams:

\[\log p(\theta|D,B,N)=\log q(\theta)-\frac{\lambda_{B}}{|B|}\sum_{b=1}^{|B|}H(y_ {b}|x_{b})-\frac{\lambda_{N}}{|N|}\sum_{n=1}^{|N|}H(y_{n}|x_{n}). \tag{8}\]

Figure 8: Effect of hyperparameters on the model accuracy on CIFAR10-C for 15 types of corruptions under five scenarios: Benign, Near, Far, Attack, and Noise. Averaged over three different random seeds.

With Equation 7 and Equation 8, we can now derive model parameter variations caused by noisy test samples:

\[\log p(\theta|D,B)-\log p(\theta|D,B,N)=\frac{\lambda_{N}}{|N|}\sum_{n=1}^{M}H(y_ {n}|x_{n}). \tag{9}\]

Equation 9 implies that the (1) model adapted only from benign data and (2) model adapted with both benign and noisy data differ by the amount of the average entropy of noisy samples. This also suggests that a high entropy from severe noisy samples would result in a significant model drift in adaptation (i.e., model corruption).

### Comparison with previous TTA methods

#### d.2.1 EATA and SAR

While SoTTA, EATA [28], and SAR [29] all leverage sample filtering strategy, the key distinction of input-wise robustness of SoTTA and EATA/SAR lies in three aspects: (1) Our high-confidence sampling strategy in SoTTA aims to filter noisy samples by utilizing only the samples with high confidence, while both EATA and SAR use a different approach that excludes a few high-entropy samples, particularly during the early adaptation stage. In our preliminary study, we found that our method excludes 99.98% of the noisy samples, whereas EATA and SAR exclude 33.55% of such samples. (2) While EATA and SAR adapt to every incoming low-entropy sample, SoTTA leverages a uniform-class memory management approach to prevent overfitting. As shown in Figure 5b, noisy samples often lead to imbalanced class predictions, and these skewed distributions could lead to an undesirable bias in \(p(y)\) and thus might negatively impact TTA objectives, such as entropy minimization. The ablation study in Table 10 shows the effectiveness of uniform sampling with a 3.9%p accuracy improvement. (3) EATA and SAR reset the memory buffer and restart the sample collection process for each adaptation. This strategy is susceptible to overfitting due to a smaller number of samples used for adaptation and the temporal distribution drift of the samples. In contrast, our continual memory management approach effectively mitigates this issue by retaining high-confidence uniform-class samples in the memory, as shown in Table 10.

We acknowledge that both SoTTA and SAR utilize sharpness-aware minimization proposed by Foret et al. [4]. However, we clarify that the motivation behind using SAM is different. While SAR intends to avoid model collapse when exposed to samples with large gradients, we aim to enhance the model's robustness to noisy samples with high confidence scores. As illustrated in Figure 6, we observed that entropy-sharpness minimization effectively prevents the model from overfitting to noisy samples. As a result, while our algorithm led to marginal performance degradation in noisy settings (82.2% \(\rightarrow\) 80.0% for Noise), EATA and SAR showed significant degradation (EATA 82.4% \(\rightarrow\) 36.0% for Noise; SAR 78.3% \(\rightarrow\) 58.3% for Noise).

#### d.2.2 RoTTA

Regarding our high-confidence uniform sampling technique, RoTTA [44] could be compared. First of all, RoTTA's objective is different from ours; RoTTA focused on temporal distribution changes of test streams without considering noisy samples. Similar to SoTTA, RoTTA's memory bank maintains recent high-confidence samples. However, RoTTA has no filtering mechanism for low-confidence samples, which makes RoTTA fail to avoid noisy samples, especially in the early stage of TTA. In contrast, our confidence-based memory management scheme effectively rejects noisy samples, and

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & Benign & Near & Far & Attack & Noise & Avg \\ \hline SoTTA (w/o High-confidence) & 82.2 ± 0.2 & 78.0 ± 0.4 & 75.9 ± 0.5 & 84.3 ± 0.1 & 77.7 ± 0.7 & 79.6 ± 0.2 \\ SoTTA (w/o Uniform-class) & **82.3 ± 0.2** & 80.9 ± 0.6 & 74.9 ± 2.4 & 83.5 ± 0.2 & 68.7 ± 7.0 & 78.0 ± 2.0 \\ SoTTA (w/o Continual) & 81.0 ± 0.5 & 79.5 ± 0.3 & 75.5 ± 1.8 & 84.4 ± 0.2 & 65.7 ± 7.0 & 77.2 ± 1.8 \\ SoTTA & 82.2 ± 0.3 & **81.4 ± 0.5** & **81.6 ± 0.6** & **84.5 ± 0.2** & **80.0 ± 1.4** & **81.9 ± 0.5** \\ \hline \hline \end{tabular}
\end{table}
Table 10: Average classification accuracy (%) and their corresponding standard deviations on ablation study of the effect of high-confidence uniform-class continual memory of SoTTA on CIFAR10-C. **Bold** numbers are the highest accuracy. Averaged over three different random seeds.

thus it prevents potential model drift from the beginning of TTA scenarios. As a result, our approach outperforms RoTTA in noisy test streams (e.g., 5.4%p better than RoTTA on CIFAR10-C).

### Comparison with out-of-distribution detection algorithms

We discussed the limitation of applying out-of-distribution detection to TTA in Section 5. Still, we are curious about the effect of applying out-of-distribution algorithms to our scenario. To this end, we conduct experiments using one of the out-of-distribution algorithms, ODIN [20], in our noisy data streams. Specifically, we filtered OOD samples detected by ODIN and performed TTA algorithms on the samples left.

Note that similar to prior studies on OOD, ODIN uses a thresholding approach to predict whether a sample is OOD. It thus requires validation data with binary labels indicating whether it is in-distribution or OOD to decide the best threshold \(\delta\). However, in TTA scenarios, validation data is not provided, which makes it difficult to apply OOD algorithms directly in our scenario. We circumvented this problem using the labeled test batches to get the best threshold. Following the original paper, we searched for the best threshold from 0.1 to 0.12 with a step size of 0.000001, which took over 20,000 times longer than the original TTA algorithm.

Table 11 shows that the impact of discarding OOD samples with ODIN is negligible, yielding only a 0.3%p improvement in the average accuracy despite a huge computation cost. Also, Figure 9 shows the high sensitivity of ODIN with respect to threshold hyperparameter \(\delta\), which implies that applying OOD in TTA is impractical.

We conclude the practical limitations of OOD detection algorithms for TTA as follows: (1) OOD methods assume that a model is fixed during test time, while a model changes continually in TTA. (2) As previously noted, most OOD algorithms require labels for validation data unavailable in TTA scenarios. Even using the same test dataset for selecting the threshold, the performance improvement was marginal. (3) Low performance possibly results from the fact that OOD detection studies are

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{Benign} & \multicolumn{3}{c}{Near} & \multicolumn{3}{c}{Far} & \multicolumn{3}{c}{Attack} & \multicolumn{3}{c}{Noise} \\ \cline{2-13} Method & w/o ODIN & w/ ODIN & w/o ODIN & w/ ODIN & w/ ODIN & w/ ODIN & w/ ODIN & w/ ODIN & w/ ODIN & w/ ODIN \\ \hline Source & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 & 57.7 ± 0.1 \\ BN stats [27] & 78.2 ± 0.3 & 78.2 ± 0.3 & 76.5 ± 0.4 & 76.5 ± 0.4 & 75.4 ± 0.3 & **75.9 ± 0.4** & 55.8 ± 1.4 & 55.8 ± 1.4 & 55.9 ± 0.8 & 56.7 ± 0.9 \\ PL [17] & 78.4 ± 0.3 & **78.8** ± 0.5 & 73.1 ± 0.3 & **74.3 ± 0.6** & 71.3 ± 0.8 & 66.5 ± 1.1 & 66.5 ± 1.1 & 52.1 ± 0.4 & 52.1 ± 0.4 \\ TENT [38] & 81.5 ± 1.0 & 81.5 ± 10.8 & 76.1 ± 0.6 & 76.1 ± 0.6 & 73.5 ± 1.4 & 77.3 ± 0.9 & **69.1 ± 0.1** & 54.4 ± 0.3 & **56.2 ± 0.6** \\ LAME [1] & 56.1 ± 0.3 & 56.1 ± 0.3 & 56.7 ± 0.5 & 56.7 ± 0.8 & 55.7 ± 0.4 & 55.7 ± 0.4 & 56.2 ± 0.8 & 56.2 ± 0.8 & 54.9 ± 0.8 & **55.2 ± 0.7** \\ CoTTA [39] & 82.2 ± 0.3 & 82.2 ± 0.3 & 78.2 ± 0.3 & 78.2 ± 0.4 & 73.6 ± 0.9 & 69.6 ± 1.3 & 69.6 ± 1.3 & 57.8 ± 0.8 & **62.0 ± 1.3** \\ ETA [28] & 82.4 ± 0.3 & 82.4 ± 0.3 & 63.9 ± 0.4 & **69.2 ± 0.4** & 56.3 ± 0.3 & **59.9 ± 0.6 & 70.9 ± 0.7 & 70.9 ± 0.7 & 36.0 ± 0.8 & 58.0 ± 0.1 \\ SAR [29] & 78.4 ± 0.7 & 78.4 ± 0.7 & 72.8 ± 0.2 & 72.8 ± 0.2 & 75.7 ± 0.3 & **76.0 ± 0.1** & 56.2 ± 1.8 & 56.2 ± 1.4 & 58.7 ± 0.3 & 58.7 ± 0.3 \\ RoTTA [44] & 75.3 ± 0.7 & 75.3 ± 0.7 & 77.5 ± 0.5 & 77.5 ± 0.7 & 77.0 ± 0.9 & 77.4 ± 0.8 & 78.4 ± 0.8 & 73.5 ± 0.5 & 73.5 ± 0.5 \\ SoTTA & 82.1 ± 0.4 & 82.1 ± 0.4 & 81.6 ± 0.4 & 81.6 ± 0.4 & 81.7 ± 0.8 & **82.0 ± 0.8** & **84.5 ± 0.3** & **84.5 ± 0.3** & 81.5 ± 1.2 & 81.5 ± 1.2 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Average classification accuracy (%) of ODIN+TTA on CIFAR10-C. **Bold** numbers are the accuracy with improvement from normal TTAs. Averaged over three different random seeds.

Figure 9: Effect of OOD threshold \(\delta\) on classification accuracy (%) of ODIN+TENT on CIFAR10-C. Averaged over three different random seeds.

built on the condition that training and test domains are the same, which differs from TTA's scenario. These collectively make it difficult to apply OOD detection studies directly to TTA scenarios.

### Applying to other domains

While this study primarily focuses on classification tasks, there are other tasks where test-time adaptation would be useful. Here we discuss the applicability of SoTTA to (1) image segmentation and (2) object detection, which are crucial in autonomous driving scenarios.

For image segmentation, when noisy objects are present in the input, the model might produce noisy predictions on those pixels, leading to detrimental results. Extending SoTTA to operate at the pixel level would allow it to be compatible with the segmentation task while minimizing the negative influences of those noisy pixels on model predictions in test-time adaptation scenarios.

Similarly, SoTTA could be tailored to object detection's classification (recognition) task. For example, in the context of the YOLO framework [32], SoTTA could filter and store grids with high confidence for test-time adaptation, enhancing detection accuracy. However, our current approach must address the localization task (bounding box regression) during test-time adaptation. Implementing this feature is non-trivial and would require careful consideration and potential redesign of certain aspects of our methodology. Accurately localizing bounding boxes during test-time adaptation presents an exciting avenue for future research.

## Appendix E License of assets

DatasetsCIFAR10/CIFAR100 (MIT License), CIFAR10-C/CIFAR100-C (Creative Commons Attribution 4.0 International), ImageNet-C (Apache 2.0), and MNIST (CC-BY-NC-SA 3.0).

CodesTorchvision for ResNet18 (Apache 2.0), the official repository of CoTTA (MIT License), the official repository of TENT (MIT License), the official repository of LAME (CC BY-NC-SA 4.0), the official repository of EATA (MIT License), the official repository of SAR (BSD 3-Clause License), and the official repository of RoTTA (MIT License).

Response:
## Review

### Summary

This paper addresses the challenge of test-time adaptation (TTA) in the presence of non-interest samples, proposing a method called Screening-out Test-Time Adaptation (SoTTA). The approach focuses on both input-wise and parameter-wise robustness to enhance performance under noisy conditions. The authors validate their method through comprehensive experiments on standard datasets, demonstrating significant improvements compared to existing TTA methods. Although the problem setting is novel and relevant, reviewers note concerns regarding the technical originality of the proposed methods, as they are closely related to existing techniques. Overall, the work is recognized for its practical implications and experimental results.

### Strengths

- The problem of non-interest samples in TTA is novel and practically significant.
- The proposed method, SoTTA, includes reasonable adaptations for robustness.
- The paper presents clear experiments demonstrating state-of-the-art performance.
- The presentation of the paper is generally good, with clear figures.

### Weaknesses

- The proposed methods may lack originality, drawing heavily from existing techniques like RoTTA and SAR.
- Key results are not fully presented in the main paper, particularly those related to ImageNet-C.
- There is no discussion on the selection and sensitivity of key hyperparameters, which is critical for practical applications.
- Several typographical errors are present in the manuscript.

### Questions

- Can the authors discuss the potential performance if CSTU and ESM methods were combined?
- What is the outcome when the proposed method is compared to EATA?
- Can the authors provide sensitivity analysis regarding the threshold C0?
- How do the proposed methods differ from existing techniques like RoTTA and SAR?

### Soundness

**Score:** 2

**Description:** 2 = fair: The methodology is reasonable but lacks originality and thoroughness in addressing important aspects.

### Presentation

**Score:** 3

**Description:** 3 = good: The paper is mostly well-presented with clear figures and layouts, but it contains some typographical errors.

### Contribution

**Score:** 2

**Description:** 2 = fair: The contribution is relevant, but the originality of the methods used is questioned due to similarities with existing approaches.

### Rating

**Score:** 5

**Description:** 5 = borderline accept: The paper is technically solid with significant contributions, though it requires some improvements and clarification on methodological originality.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents a relevant and novel problem in the field of test-time adaptation and proposes a method with promising experimental results. However, concerns regarding technical originality and thoroughness in presenting results and methodology exist. The decision to accept aligns with the majority of reviewer feedback, recognizing the potential impact of the work while suggesting areas for improvement.