# Domain-Specific Block Selection and Paired-View Pseudo-Labeling for
Online Test-Time Adaptation

Yeonguk Yu, Sungho Shin, Seunghyeok Back, Minhwan Ko, Sangjun Noh, and Kyoobin
Lee  Gwangju Institute of Science and Technology
{yeon_guk, hogili89, shback, mhko1998<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Test-time adaptation (TTA) aims to adapt a pre-trained model to a new test
domain without access to source data after deployment. Existing approaches
typically rely on self-training with pseudo-labels since ground-truth cannot
be obtained from test data. Although the quality of pseudo labels is important
for stable and accurate long-term adaptation, it has not been previously
addressed. In this work, we propose DPLOT, a simple yet effective TTA
framework that consists of two components: (1) domain-specific block selection
and (2) pseudo-label generation using paired-view images. Specifically, we
select blocks that involve domain-specific feature extraction and train these
blocks by entropy minimization. After blocks are adjusted for current test
domain, we generate pseudo-labels by averaging given test images and
corresponding flipped counterparts. By simply using flip augmentation, we
prevent a decrease in the quality of the pseudo-labels, which can be caused by
the domain gap resulting from strong augmentation. Our experimental results
demonstrate that DPLOT outperforms previous TTA methods in CIFAR10-C,
CIFAR100-C, and ImageNet-C benchmarks, reducing error by up to 5.4%, 9.1%, and
2.9%, respectively. Also, we provide an extensive analysis to demonstrate
effectiveness of our framework. Code is available at https://github.com/gist-
ailab/domain-specific-block-selection-and-paired-view-pseudo-labeling-for-
online-TTA.

##  1 Introduction

Deep neural networks achieve remarkable performance when the training and
target data originate from the same domain [39, 36]. In contrast, deployed
models perform poorly if domain shifts exists between source training data and
target test data [35, 12]. For example, a pre-trained image classification
model may suffer this phenomenon for the given corrupted images due to sensor
degradation, weather change, and other reasons [37, 15]. Various studies have
addressed the domain shift issue [19, 55, 29]. Recently, test-time adaptation
(TTA), which aims to improve the model performance on target domain without
access to the source data during the inference stage, has received attention
because of its practicality and applicability [43, 44, 32].

Figure 1: Results of the proposed framework for online test-time adaptation
(Orange). We evaluate average error rates of the WideResNet40 and ResNext-29
architectures for the CIFAR100-C gradual setting benchmark using competitive
test-time adaptation methods. In the gradual setting, the networks should
adapt to continually changing corruption domains (135 changes in total).

In online TTA, the objective is to simultaneously make prediction and
adaptation using a source pre-trained model for the given test data. Existing
TTA methods typically rely on self-training with pseudo labels, such as
entropy minimization [13, 42] and consistency regularization [41, 40]. Entropy
minimization trains the model by self-generated pseudo-labels, whereas
consistency regularization uses pseudo-labels generated by a teacher model.
These methods have demonstrated excellent performance with short-term test
sequence in the stationary environment [43].

Under the continually changing domain in non-stationary environments [30, 44],
self-training with pseudo-labels can lead to error accumulation, gradually
degrading the quality of pseudo-labels [44]. To alleviate the issue,
stochastic restoration (i.e., reset the model to the source pre-trained
weight) and augmentation-averaged pseudo-label (i.e., various hard
augmentations including color jitter) was proposed by [44]. In addition,
Döbler _et al_. [8] combined symmetric cross-entropy [46] and contrastive loss
for stabilizing the pseudo-labeling without restoration. However, previous
methods lacks an accurate approach to generate pseudo-labels from the teacher,
a critical aspect for stable long-term test-time adaptation.

In this study, we address accurate pseudo-label generation in underlying
assumption that images from source domain and target domain share domain-
invariant features for a task, regardless of shifted domain-specific features
caused by corruptions. We consider the domain-invariant feature to be a high-
level feature useful for the task and the domain-specific feature to be a low-
level feature with no concern for the task as termed in [47, 49, 48]. To
achieve precise adaptation under this assumption, we propose Domain-specific
block selection and paired-view Pseudo-Labeling for Online Test-Time
adaptation (DPLOT), which is composed of two core components: (1) domain-
specific block selection and (2) pseudo-label generation using paired-view
images. The block selection method identifies blocks involve in domain-
specific feature extraction (termed domain-specific block) using augmented
source training data. These domain-specific blocks are then fine-tuned through
entropy minimization during the test-time phase. Then, we generate the pseudo-
label from the teacher by averaging predictions for the given test image and
its horizontally flipped counterpart to update all model parameters. This is
motivated by the fact that the teacher model’s domain-specific blocks are
adjusted for the current domain, and hard augmentation may generate another
domain gap, which may lead to degraded pseudo-labels. Consequently, our
framework provides strong adaptation performance for the model as shown in
Figure 1.

In summary, the main contributions are as follows:

  * •

We propose the DPLOT, which consists of domain-specific block selection and
paired-view pseudo labeling for long-term online test-time adaptation.

  * •

We compare the proposed method with other online TTA methods in both continual
and gradual settings benchmarks, and our framework outperforms other methods.

  * •

We provide a wide range of analyses that lead to an improved understanding of
our framework.

##  2 Related Work

#### Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) aims to adapt a model to a target domain
using given labeled source data and unlabeled target data before model
deployment.For example, Ganin and Lempitsky [11] proposed gradient reversal
layer to force the feature extractor to produce same feature distribution for
the given source data and target data . Also, Hoffman  _et al_. [18] used
image-to-image translation to create labeled target-like source data to train
the network. French  _et al_. [9] used self-ensembling with the mean teacher
to minimize the difference between the student’s prediction for the augmented
test data and the teacher’s prediction for the test data. Kundu  _et al_. [25]
proposed class-incremental method without using source training data.
Recently, Hoyer  _et al_. [19] proposed the adaptation framework based on
masked image consistency, where the model is forced to produce the same
prediction for the given target data and corresponding masked data. Also,
Prasanna  _et al_. [34] proposed the continual domain adaptation method based
on pruning-aided weight modulation to reduce catastrophic forgetting. Since
both UDA and TTA aim to performance improvement in target domain, the methods
from UDA can be considered. Our method also uses the mean teacher as in [19]
to reduce the prediction difference for adapting the network to the target
domain but without using source data.

#### Test-Time Adaptation

Test-time adaptation (TTA), which only requires target data for adaptation
unlike UDA, has gained increasing attention. As in [27], (online) TTA can be
categorized into batch normalization (BN) calibration [31, 53], entropy
minimization [43, 32, 33, 4], and consistency regularization [44, 8, 45].
Specifically, Mizra  _et al_. [31] proposed dynamic unsupervised adaptation
(DUA), which adapts the statistics of the BN layers[20] to remove degradation
caused by BN[26, 10]. On the other hand, updating affine parameters of BN
layers has demonstrated improved adaptation performance [43, 32, 33, 4]. Niu
_et al_. [33] demonstrated that using batch-independent normalization methods
like layer norm [1] and group norm [50] instead of using batch-dependent BN is
helpful for stable entropy minimization. Also, they proposed sharpness-aware
entropy minimization, which filters out unreliable samples by their gradients,
based on the observation that large gradient samples lead to model collapse.
Similar to our work, Choi  _et al_. [4] proposed the framework that updates
model parameters by entropy minimization, differently depending on their
sensitivity of distribution shift. Simultaneously, consistency regularization-
based TTA frameworks have been investigated. Wang  _et al_. [44] proposed TTA
in a continually changing domain by using augmentation-averaged pseudo-target
from the mean teacher. Also, Chen  _et al_. [3] proposed to use contrastive
learning in TTA for learning better representation in the target domain.
Moreover, robust mean teacher (RMT) is proposed [8], which showed state-of-
the-art performance by using various techniques such as symmetric cross-
entropy and source replay. Previous entropy minimization methods [43, 32, 33]
update BN layer of the model which can modify domain-invariant feature
extraction. In this work, we propose block selection method to prevent the
modification of domain-invariant feature extraction by updating blocks that
involve in domain-specific feature extraction. Also, we propose to generate
pseudo-labels from the teacher only using simple flip augmentation to improve
the quality of labels. Consequently, our DPLOT uses both entropy minimization
and consistency regularization for reliable long-term adaptation.

##  3 Method

We introduces DPLOT, a simple yet effective online TTA framework. First, we
describe the overview of our framework in Section 3.1. Second, we describe the
block selection method before deployment in Section 3.2. Lastly, we introduce
how our framework adapts the model for the given unlabeled test data after
deployment in Section 3.3.

###  3.1 Overview of DPLOT’s components

The components within our framework can be categorized based on whether they
are used before or after deployment. Domain-specific block selection is
conducted before deployment, while our adaptation method as shown in Figure 2,
involving entropy minimization on the selected blocks and the use of paired-
view pseudo-labels for updating all model parameters, is performed after
deployment.

The domain-specific block selection is proposed under the assumption that the
domain-invariant feature of test image remains consistent with the source
domain, while the domain-specific feature changes. Therefore, the domain-
specific feature extraction of the model should be adjusted to bridge the gap
between the source domain and target domain caused by different feature
statistics [26, 2, 47]. We evaluate each block by measuring cosine similarity
between prototype features before and after the entropy minimization using
Gaussian noise-added source data. Subsequently, we select the blocks that
maintain high similarity, indicating that the blocks do not involve in domain-
specific feature extraction (i.e., involves in domain-invariant feature
extraction). During test-time, the selected blocks are updated by minimizing
entropy for the given test data to adjust the domain-specific feature
extraction to the current corruption. By selecting domain-specific blocks, we
can adjust domain-specific feature extraction without disrupting domain-
invariant feature extraction unlike previous method [43], where all batch
normalization [20] layers are updated.

We provide pseudo-labels generated by the exponential moving average (EMA)
teacher [41] (i.e.,
$\theta^{\prime}_{t+1}=\alpha\theta^{\prime}_{t}+(1-\alpha)\theta_{t+1}$) to
the model for further adjusting all parameters on the target domain. This is
based on insight that the high-level feature in the test image can be affected
by domain shifts, and all parameters should be adjusted to increase adaptation
performance. In contrast to previous approach [44], where hard augmentations
(e.g., color jitter, Gaussian noise, blur, and random pad-crop) are used for
generating pseudo-labels, we only use horizontal flip augmentation. Since the
teacher’s domain-specific extraction is adjusted on the target domain by
entropy minimization and even a small domain gap between the test and
augmented images can reduce the teacher’s accuracy, the horizontal flip is
well suited for generating pseudo-labels from the teacher.

Figure 2: Illustration of our proposed test-time adaptation using entropy
minimization and paired-view consistency. During test-time, the current test
and corresponding flipped images are given to the student model and EMA
teacher model. Entropy minimization is performed to update the parameters of
selected blocks (yellow arrow), while all parameters are updated to minimize
the difference between student output and the averaged EMA model’s prediction
(blue arrow).

###  3.2 Block selection before deployment

We consider a pre-trained neural network $\theta$, comprising a feature
extractor and a classifier. We assume that the feature extractor of the
network is composed of multiple $L$ blocks (e.g., ResNet18 has 8 residual
blocks [14]). For a given a RGB image $x\in\mathbb{R}^{H\times W\times 3}$, a
feature vector $f\in\mathbb{R}^{d}$ of dimension $d$ is extracted. Then, our
objective is to select blocks that involve in domain-specific feature
extraction. To this end, we first calculate the domain-invariant feature space
using prototype vectors. The prototype vector $p_{c}\in\mathbb{R}^{d}$ is
acquired by averaging all source feature vectors belonging to the class $c$ as
follows:

$p_{c}=\frac{1}{N_{c}}\sum_{n=1}^{N_{c}}f(x^{c}_{n}\in\mathcal{X}^{S};\theta),$
(1)

where $f(\cdot),x^{c}_{n}$, and $N_{c}$ refer to the feature extraction,
$n$-th RGB image belonging to the class $c$, and number of images for class
$c$, respectively. Consequently, prototypes for all class,
$P=\\{p_{1},p_{2},...,p_{C}\\}$, are obtained.

To further select the block for adjusting domain-specific feature extraction
during test-time, we measure the similarity $s$ between the original
prototypes and modified prototypes after the block’s parameters have been
updated via entropy minimization with Gaussian noise (zero mean and 0.5
variance) added training images. We use Gaussian noise as it is common
corruption and can represent various domain shifts [28, 6]. This similarity is
calculated by averaging cosine similarity as follows:

$s_{i}=\frac{1}{C}\sum_{c=1}^{C}\frac{p_{c}\cdot
p^{\prime}_{c}}{\|p_{c}\|\|p^{\prime}_{c}\|},$ (2)

where $p^{\prime}_{c}$ denotes a single modified prototype vector for the
$c$-th class after the entropy minimization. Also, $C$ refers to the number of
classes. After computing the similarity for every block, we apply min-max
scaling to normalize the results into the range of [0, 1]. This scaling allows
us to choose threshold more generally across various architectures. A high
similarity indicates that the block is adapted on Gaussian noise added images
without modifying domain-invariant feature space (i.e., high-level feature
space). Finally, we select blocks that higher than the threshold $\gamma$. The
pseudo code of the block selection is described in Algorithm 1. In our
experiments, the threshold is set to 0.75, unless otherwise specified.

Init. : $\text{An empty list }\mathcal{S},\text{an empty list
}\mathcal{B}^{s};$

Input : Blocks of the model $\mathcal{B}=\\{b_{1},b_{2},...,b_{L}\\}$, source
data $(\mathcal{X}^{S},\mathcal{Y}^{S})$, threshold $\gamma$, pre-trained
model $\theta$;

Calculate $P$ using Eq. 1

Add noise to $\mathcal{X}^{S}$

foreach  _block $b_{i}\in\mathcal{B}$_ do

Minimize entropy for parameter of $b_{i}$ using $\mathcal{X}^{S}$

Calculate $P^{\prime}$ using Eq. 1

Calculate $s_{i}$ using Eq. 2

Append $s_{i}$ in $\mathcal{S}$

Reset $\theta$;

end foreach

Min-max scale $S$

foreach  _$(s_{i},b_{i})\in(\mathcal{S},\mathcal{B})$_ do

if  _$s_{i} >\gamma$_ then

Append $b_{i}$ in $\mathcal{B}^{s};$

end if

end foreach

Output : Selected block list $\mathcal{B}^{s}$

Algorithm 1 Domain-specific block selection

Init. : Selected Blocks $\mathcal{B}^{s}$, a pre-trained model $\theta$, a
teacher model $\theta^{\prime}$ initialized from $\theta$;

Input : For each time step $t$, current batch of data $x_{t}^{\mathcal{T}}$;

1: Horizontally flip $x_{t}^{\mathcal{T}}$ and get pseudo-label from
$\theta^{\prime}_{t}$ by Eq. 4 and Eq. 5

2: Update $\mathcal{B}^{s}$ of $\theta_{t}$ by entropy minimization loss
$\mathcal{L}_{e}$ in Eq. 3 using both $x_{t}^{\mathcal{T}}$ and
$\tilde{x}_{t}^{\mathcal{T}}$

3: Update $\theta_{t}$ by paired-view consistency loss $\mathcal{L}_{p}c$ in
Eq. 6 using both $x_{t}^{\mathcal{T}}$ and $\tilde{x}_{t}^{\mathcal{T}}$

4: Update $\theta^{\prime}_{t}$ by moving average of $\theta_{t}$

Output : Ensemble prediction, Updated model $\theta_{t+1}$, Updated teacher
$\theta^{\prime}_{t+1}$

Algorithm 2 Adaptation process after deployment

###  3.3 Test-time adaptation after deployment

#### Entropy minimization

Entropy minimization is performed for the given target data at current time
$x_{t}^{T}$ to update the selected blocks $\mathcal{B}^{s}$ of model $\theta$.
Following other entropy minimization-based methods [43, 32, 33, 4], we use
Shannon Entropy [38] as follows:

$\mathcal{L}_{e}=-\sum_{c}\hat{y}_{c}\log\hat{y}_{c},$ (3)

where $\hat{y}_{c}$ represents the output probability for the $c$-th class
using the model $\theta$. By minimizing entropy on test batch
$x^{\mathcal{T}}_{t}$, the model is trained to push decision boundaries toward
low-density region in prediction space [42]. As a result, the model is forced
to acquire discriminative high-level features from current target domain by
adjusting domain-specific feature extraction.

#### Paired-view consistency

Domain-specific feature extraction of the model is adjusted to the current
corruption using entropy minimization with selected domain-specific blocks. To
leverage all parameters, the consistency regularization between the model
$\theta$ and the moving average teacher $\theta^{\prime}$, which makes the
training stable [41, 44, 8], is used. Specifically, we use horizontal flip and
moving average teacher [41] to generate pseudo-labels for the given test and
corresponding flipped images as follows:

$\hat{y}^{\prime}=g_{\theta^{\prime}}(x_{t}^{\mathcal{T}}),$ (4)
$\tilde{y}^{\prime}=g_{\theta^{\prime}}(\tilde{x}_{t}^{\mathcal{T}}),$ (5)

where $g_{\theta^{\prime}}(\cdot)$ refers to the teacher’s prediction of the
given input. Also, $x$ and $\tilde{x}$ refer to the original input and
horizontally flipped input. The pseudo-label $\bar{y}^{\prime}$ is calculated
by averaging $\hat{y}^{\prime}$ and $\tilde{y}^{\prime}$. Subsequently, all
parameters of the model $\theta$ are updated by the symmetric cross-entropy
[8, 46] between the model and teacher predictions:

$\mathcal{L}_{pc}=\mathcal{L}_{sce}(\hat{y},\bar{y}^{\prime})+\mathcal{L}_{sce}(\tilde{y},\bar{y}^{\prime}),$
(6)
$\mathcal{L}_{sce}(a,b)=\frac{1}{2}\cdot(\mathcal{L}_{ce}(a,b)+\mathcal{L}_{ce}(b,a)),$
(7)

where $\mathcal{L}_{sce}$ and $\tilde{y}$ refer to the symmetric cross-entropy
and the model’s prediction for the flipped input, respectively. Also, $L_{ce}$
refers to the standard cross-entropy loss. We use the symmetric cross-entropy
since it is known to be robust to noisy labels [8, 46]. After adaptation, we
ensemble the predictions of both models by adding the model’s and teacher’s
logits as [8] for better performance. Our method during test-time is
summarized in Algorithm 2.

##  4 Experiments

#### Setup

We evaluate our framework on CIFAR10-C, CIFAR100-C, and ImageNet-C, designed
to benchmark the robustness of classification networks [15]. CIFAR dataset
contains 10,000 and ImageNet dataset contains 50,000 test images for each of
the 15 corruptions with 5 levels of severity. For the experiments, we use a
network pre-trained on the clean training set of CIFAR [24] and ImageNet [7].
For CIFAR10, we use WRN28-10 [52], WRN40-2A [16], ResNet-18A [22]. For
CIFAR100, we use WRN40-2A [16] and ResNext-29 [51]. For ImageNet, we use
ResNet50 [14] and ResNet50A [17]. Architectures named with ’A’ refers to the
networks trained to be robust against corruption (e.g., AugMix [16]); specific
details are described in supplementary materials.

We evaluate our method in two different settings. First, we consider the
continual setting introduced by [44]. Unlike the basic TTA setting, where
evaluation is conducted for each corruption individually, the model is adapted
to a sequence of test domains in an online fashion under the largest
corruption severity level 5 (total 15 shifts). Second, we consider the gradual
setting, as introduced by [30] where the corruption severity level changes as
follows: 1 $\rightarrow$ 2 $\rightarrow\cdots\rightarrow
5\rightarrow\cdots\rightarrow$ 2 $\rightarrow$ 1 (total 135 shifts). This
setting is motivated by the fact that domain shift does not occur abruptly but
changes rather smoothly.

Also, we follow the implementation setting of RMT [8]. Specifically, the batch
size during test-time is set to 200 and 64 for CIFAR and ImageNet-C,
respectively. We use an Adam [21] optimizer with a learning rate of 1e-3 and
1e-4 for entropy minimization to domain-specific blocks and paired-view
consistency to all blocks, respectively. Warm-up is conducted before
deployment, as done in [8], and we use pre-trained weights provided by
RobustBench [5].

#### Baselines

We compare our method with various source-free TTA baselines. Also, BN-1
refers to method that recalculates the BN statistics using the test batch.
TENT [43], EATA [32], and SAR [33] are entropy minimization-based methods that
update batch normalization layer weights to minimize the entropy of current
predictions. AdaContrast [3] relies on contrastive learning principles,
combining contrastive learning and pseudo-labeling to enable discriminative
feature learning for TTA. CoTTA[44] uses a teacher’s augmentation-averaged
pseudo-label for training and stochastic restore to mitigate error
accumulation. RMT [8] uses the symmetric cross-entropy, which reduces effect
of noisy label, for consistency regularization with pseudo-labels generated by
a teacher and contrastive learning to pull the test feature space closer to
the source domain. Moreover, RMT utilizes source replays during test-time to
keep source knowledge. However, it is worth noting that we do not use source
replays as it is not source-free during test-time.

Method | CIFAR100-C | ImageNet-C
---|---|---
ResNext-29A | WRN40-2A | ResNet-50 | ResNet-50A
Source only | 46.4† | 45.4 | 82.0† | 67.5
BN-1 | 35.4† | 39.3 | 68.6† | 53.8
TENT-cont. | 60.9† | 37.5 | 62.6† | 49.6
AdaContrast | 33.4† | 37.1 | 65.5† | 50.9
CoTTA | 32.5† | 38.2 | 62.7† | 47.8
EATA | 32.3 | 35.7 | 58.8 | 46.3
SAR | 31.9 | 35.3 | 61.9 | 49.3
RMT | 29.0† | 34.3 | 59.8† | 46.9
DPLOT (ours) | 27.8 | 31.8 | 60.2 | 44.6

Table 1: Averaged classification error rate (%) for the CIFAR100-C and
ImageNet-C benchmarks with the continual setting. The error rates are averaged
for the given 15 corruption. † indicates that the result is reported by [8].

TIME | t$\longrightarrow$ |
---|---|---
Method |  Gaussian |  shot |  impulse |  defocus |  glass |  motion |  zoom |  zoom |  frost |  fog |  brightness |  contrast |  elastic |  pixelate |  jpeg | Mean
WRN28-10 | Source only† | 72.3 | 65.7 | 72.9 | 46.9 | 54.3 | 34.8 | 42.0 | 25.1 | 41.3 | 26.0 | 9.3 | 46.7 | 26.6 | 58.5 | 30.3 | 43.5
BN-1† | 28.1 | 26.1 | 36.3 | 12.8 | 35.3 | 14.2 | 12.1 | 17.3 | 17.4 | 15.3 | 8.4 | 12.6 | 23.8 | 19.7 | 27.3 | 20.4
TENT-cont.† | 24.8 | 20.6 | 28.6 | 14.4 | 31.1 | 16.5 | 14.1 | 19.1 | 18.6 | 18.6 | 12.2 | 20.3 | 25.7 | 20.8 | 24.9 | 20.7
AdaContrast† | 29.1 | 22.5 | 30.0 | 14.0 | 32.7 | 14.1 | 12.0 | 16.6 | 14.9 | 14.4 | 8.1 | 10.0 | 21.9 | 17.7 | 20.0 | 18.5
CoTTA† | 24.3 | 21.3 | 26.6 | 11.6 | 27.6 | 12.2 | 10.3 | 14.8 | 14.1 | 12.4 | 7.5 | 10.6 | 18.3 | 13.4 | 17.3 | 16.2
EATA | 24.3 | 19.3 | 27.6 | 12.6 | 28.6 | 14.4 | 12.0 | 15.9 | 14.6 | 15.4 | 9.6 | 13.3 | 20.6 | 16.3 | 21.8 | 17.8
SAR | 28.3 | 26.0 | 35.8 | 12.7 | 34.6 | 13.9 | 12.0 | 17.5 | 17.6 | 14.9 | 8.2 | 13.0 | 23.5 | 19.5 | 27.2 | 20.3
RMT† | 21.9 | 18.6 | 24.1 | 10.8 | 23.6 | 12.0 | 10.4 | 13.0 | 12.4 | 11.4 | 8.3 | 10.1 | 15.2 | 11.3 | 14.6 | 14.5
DPLOT (ours) | 19.4 | 16.5 | 22.5 | 10.0 | 23.7 | 11.1 | 9.6 | 12.3 | 11.9 | 10.7 | 7.7 | 9.8 | 15.5 | 10.8 | 13.9 | 13.7
WRN40-2A | Source only | 28.8 | 22.9 | 26.2 | 9.5 | 20.6 | 10.6 | 9.3 | 14.2 | 15.3 | 17.5 | 7.6 | 20.9 | 14.8 | 41.3 | 14.7 | 18.3
BN-1 | 18.4 | 16.1 | 22.3 | 9.0 | 22.1 | 10.6 | 9.7 | 13.2 | 13.2 | 15.3 | 7.8 | 12.1 | 16.3 | 14.9 | 17.2 | 14.5
TENT-cont. | 15.0 | 12.1 | 16.8 | 9.5 | 18.0 | 11.7 | 9.9 | 11.8 | 11.4 | 13.7 | 9.3 | 11.4 | 16.8 | 13.1 | 19.8 | 13.4
AdaContrast | 16.2 | 12.6 | 16.9 | 8.3 | 18.1 | 10.0 | 8.4 | 10.7 | 9.8 | 12.0 | 7.1 | 8.4 | 14.2 | 12.1 | 13.8 | 11.9
CoTTA | 15.4 | 13.5 | 16.3 | 9.1 | 17.8 | 10.2 | 8.9 | 11.9 | 11.3 | 14.7 | 7.1 | 15.0 | 13.8 | 10.7 | 13.3 | 12.6
EATA | 15.3 | 11.7 | 16.6 | 9.0 | 17.3 | 10.8 | 9.1 | 11.4 | 10.6 | 13.5 | 8.8 | 10.6 | 15.5 | 11.7 | 16.4 | 12.5
SAR | 18.1 | 15.9 | 20.5 | 9.0 | 20.9 | 10.6 | 9.7 | 13.2 | 13.3 | 15.2 | 7.8 | 12.1 | 16.2 | 14.9 | 17.1 | 14.3
RMT | 15.3 | 12.5 | 15.4 | 8.7 | 15.8 | 9.6 | 8.1 | 9.7 | 9.6 | 10.4 | 7.2 | 9.9 | 11.3 | 8.8 | 11.4 | 10.9
DPLOT (ours) | 12.4 | 10.5 | 13.9 | 7.8 | 14.9 | 9.1 | 8.0 | 9.7 | 9.0 | 9.9 | 7.2 | 8.3 | 11.6 | 8.8 | 11.7 | 10.2
ResNet-18A | Source only | 20.2 | 17.5 | 29.3 | 8.8 | 21.7 | 10.5 | 8.7 | 13.5 | 13.5 | 21.6 | 7.2 | 34.9 | 14.3 | 17.1 | 11.8 | 16.7
BN-1 | 14.9 | 13.4 | 20.1 | 9.1 | 22.0 | 10.6 | 9.9 | 13.5 | 13.7 | 16.7 | 8.6 | 12.8 | 16.7 | 12.5 | 15.1 | 14.0
TENT-cont. | 13.1 | 11.4 | 17.7 | 9.2 | 19.8 | 12.3 | 10.9 | 13.5 | 12.8 | 16.6 | 10.4 | 11.4 | 16.3 | 12.2 | 16.2 | 13.6
AdaContrast | 13.2 | 11.2 | 16.0 | 8.5 | 17.9 | 9.8 | 8.5 | 11.2 | 9.5 | 13.5 | 6.9 | 8.2 | 13.6 | 10.1 | 10.9 | 11.3
CoTTA | 13.6 | 11.9 | 15.7 | 8.6 | 17.2 | 9.3 | 8.5 | 11.3 | 11.2 | 13.9 | 7.4 | 11.0 | 12.7 | 9.6 | 11.1 | 11.5
EATA | 13.0 | 10.9 | 16.1 | 8.5 | 17.1 | 10.0 | 8.7 | 10.6 | 10.2 | 13.9 | 7.9 | 9.5 | 14.5 | 10.5 | 13.3 | 11.6
SAR | 14.9 | 13.4 | 20.0 | 9.1 | 21.3 | 10.6 | 9.9 | 13.5 | 13.7 | 16.7 | 8.6 | 12.8 | 16.7 | 12.5 | 15.1 | 13.9
RMT | 13.3 | 10.9 | 14.9 | 8.4 | 15.1 | 9.5 | 7.9 | 9.5 | 9.6 | 10.2 | 7.5 | 9.0 | 10.9 | 8.5 | 9.7 | 10.3
DPLOT (ours) | 10.3 | 9.1 | 13.2 | 7.2 | 14.0 | 7.8 | 6.7 | 8.1 | 7.6 | 8.9 | 5.8 | 6.5 | 9.7 | 7.0 | 8.1 | 8.7

Table 2: Classification error rate (%) for the continual CIFAR10-C benchmark;
the network is trained on clean CIFAR10 and evaluated on continually given
corrupted test data. We evaluate our framework with various models: WRN28-10,
WRN40-2A, and ResNet18A. The results are averaged over five runs. Also, the
best result is indicated in bold. † indicates that the result is reported by
[8].

| CIFAR10-C | CIFAR100-C | ImageNet-C
---|---|---|---
Method | WRN28-10 | WRN40-2A | ResNet-18A | ResNext-29A | WRN40-2A | ResNet-50 | ResNet-50A
Source only | 24.7 / 43.5† | 10.4 / 18.3 | 9.7 / 16.7 | 33.6 / 46.4† | 34.7 / 46.7 | 58.4 / 82.0† | 44.9 / 67.2
BN-1 | 13.7 / 20.4† | 10.5 / 14.5 | 10.5 / 14.0 | 29.9 / 35.4† | 33.7 / 39.3 | 48.3 / 68.6† | 39.3 / 54.8
TENT-cont. | 20.4 / 25.1† | 15.0 / 18.2 | 20.0 / 22.9 | 74.8 / 75.9† | 63.5 / 65.8 | 46.4 / 58.9† | 38.5 / 47.0
AdaContrast | 12.1 / 15.8† | 8.9 / 10.9 | 8.2 / 10.0 | 33.0 / 35.9† | 35.9 / 39.1 | 66.3 / 72.6† | 56.7 / 61.5
CoTTA | 10.9 / 14.2† | 8.6 / 11.3 | 8.0 / 9.7 | 26.3 / 28.3† | 32.8 / 37.2 | 38.8 / 43.1† | 32.0 / 33.8
EATA | 16.0 / 20.6 | 11.7 / 14.5 | 11.4 / 13.9 | 32.0 / 34.4 | 33.1 / 36.8 | 40.7 / 49.7 | 36.0 / 41.2
SAR | 13.6 / 20.3 | 8.7 / 11.4 | 7.6 / 10.0 | 28.7 / 31.9 | 30.7 / 34.5 | 42.8 / 55.8 | 36.5 / 45.7
RMT | 9.3 / 10.4† | 7.7 / 8.5 | 11.7 / 12.5 | 26.4 / 26.9† | 31.1 / 32.1 | 39.3 / 41.5† | 33.5 / 34.4
DPLOT (ours) |  8.8 / 10.4 | 7.2 / 8.0 | 6.3 / 7.0 | 23.9 / 25.0 | 27.3 / 28.6 | 37.2 / 40.2 | 30.8 / 31.6

Table 3: Classification error rate (%) for CIFAR10-C, CIFAR100-C, and
ImageNet-C benchmarks with the gradual setting. Error rates are separately
reported by averaging over all severity levels and averaging only over the
highest severity level 5 (@level 1-5 / @level 5). † indicates that the result
is reported by [8].

###  4.1 Continual setting benchmark

First we evaluate our TTA framework on continual setting benchmarks. In Table
1, we provide the averaged classification error rates for CIFAR100-C and
ImageNet-C benchmarks in the continual setting, considering 15 different
corruptions. Our framework outperforms other methods for CIFAR100-C and
ImageNet-C benchmarks except when using ResNet-50. In addition, we provide
full comparison result of our framework with other TTA frameworks on CIFAR10-C
in Table 2. Our framework achieves state-of-the-art performances,
outperforming the best baseline by 5.5%, 6.4%, and 15.5% in mean error rate
when using WRN28-10, WRN40-2A, and ResNet-18A, respectively.

###  4.2 Gradual setting benchmark

In Table 3, we report the average error rate across all severity levels and
specfically with respect to level 5 in gradual setting benchmarks. We have
following observations. First, TENT suffers from the error accumulation as
observed in [32, 23] for gradual setting benchmarks due to more frequent
updates. For example, TENT has an increased mean error rate compared to
continual setting (e.g., 60.9 $\rightarrow$ 74.8 when using ResNext-29A).
Second, entropy minimization-based methods (TENT, EATA, and SAR) tend to have
degraded performance compared to consistency regularization-based methods
(CoTTA and RMT) because the self-generated pseudo-label is more susceptible to
error accumulation due to model collapse and forgetting [32, 33]. Lastly, our
framework consistently outperforms other competitive methods across different
architectures and datasets, highlighting the benefit of using domain-specific
block selection and paired-view consistency for long-time adaptation against
corruptions.

##  5 Discussion

###  5.1 Component analysis

To understand the effect of each component of our framework, we provide the
adaptation performance with various configurations in Table 4. First, we
present the performance when using all components: the paired-view
consistency, EMA teacher, and the block selection (A). If we remove paired-
view consistency using teacher-generated pseudo-labels, the performance
significantly drops (B). Furthermore, the performance slightly drops if we do
not use ensemble prediction as observed in [8] (C). Finally, it is
demonstrated that the performance significantly drops when we update BN layers
rather than domain-specific blocks by entropy minimization (D). These results
show that all components are necessary for stable long-term adaptation.

Method | CIFAR10-C | CIFAR100-C | ImageNet-C
---|---|---|---
DPLOT (A) | 8.8 / 10.4 | 23.9 / 25.0 | 30.8 / 31.6
$-$ Paired-view consistency (B) | 11.3 / 14.2 | 25.9 / 28.8 | 37.4 / 42.5
$-$ EMA teacher (C) | 12.5 / 15.8 | 25.8 / 28.5 | 37.9 / 42.8
$-$ Block selection (D) | 19.3 / 24.5 | 66.9 / 68.7 | 38.5 / 47.0

Table 4: Classification error rate (@level 5/@level 1-5) for the gradual
benchmarks with various configurations. We use WRN28-10, ResNext-29A, and
ResNet50A for CIFAR10-C, CIFAR100-C, and ImageNet-C datasets, respectively.
Note that, we gradually remove each component from DPLOT, and when none of the
components are used (D), the method is equivalent to TENT.

###  5.2 Parameter sensitivity

In Table 5, we empirically demonstrate the influence of the hyperparameter
$\gamma$ for block selection. As $\gamma$ increases, fewer blocks are selected
and vice versa. For instance, only one block is selected for entropy
minimization if we set $\gamma$ to 0.999, while all blocks except the lowest
one are selected for $\gamma$ of 0.0. We find that small $\gamma$
significantly damages the adaptation performance, but it tends to have robust
performance for the range of [0.75, 0.95]. The results demonstrate that as
blocks not involve in domain-specific feature extraction are selected to be
updated by entropy minimization, the model becomes vulnerable to the model
collapse (i.e., predicts all samples to one class [32, 33]). This
vulnerability, caused by modified domain-invariant feature space, can be
alleviated by our block selection. Also, updating selected blocks using
$\gamma$ in the range of [0.75, 0.95] improves adaptation performance compared
to updating the BN weights.

Threshold $\gamma$ | CIFAR10-C | CIFAR100-C | ImageNet-C
---|---|---|---
0.999 | 9.2 / 11.4 | 24.8 / 26.1 | 37.8 / 41.2
0.95 | 8.8 / 10.7 | 24.4 / 25.6 | 37.3 / 40.4
0.9 | 8.8 / 10.7 | 24.3 / 25.5 | 37.1 / 40.2
0.75 | 8.8 / 10.4 | 23.9 / 25.0 | 37.2 / 40.2
0.5 | 11.1 / 12.5 | 23.8 / 24.5 | 39.1 / 41.2
0.25 | 12.8 / 14.5 | 92.6 / 93.5 | 99.1 / 99.8
0.0 | 80.7 / 81.2 | 92.7 / 93.5 | 99.6 / 99.9
BatchNorm | 9.6 / 12.8 | 25.4 / 26.9 | 37.7 / 40.7

Table 5: Classification error rate (@level 5/@level 1-5) for the gradual
benchmarks with various thresholds $\gamma$ for block selection. For
CIFAR10-C, CIFAR100-C, and ImageNet-C, network architecture of WRN28-10,
ResNext-29A, and ResNet50 is used, respectively. BatchNorm indicates that
block selection is not used and BN weights are updated, as in previous methods
[43, 32, 33]. Figure 3: Illustrations of our proposed block selection results
(a, b) and classification error rate (@level 1-5) for the gradual setting
benchmark using other entropy minimization-based methods with or without block
selection (c, d). Additionally, in (a) and (b), the source accuracy after the
long-time adaptation (i.e., gradual setting benchmark) with selected blocks is
shown in a bar graph.

###  5.3 Effect of block selection

In Figure 3, we present the results of proposed block selection for WRN28-10
and ResNext-29A for CIFAR10 and CIFAR100, respectively. It is demonstrated
that the shallower blocks show high similarity between prototypes before and
after entropy minimization, while deeper blocks show lower similarity in (a,
b). These findings align with observations in [54, 4, 47] that style knowledge
(i.e., domain specific feature unrelated to the task) being predominantly
captured by shallow blocks. As expected, we find that updating blocks with
high similarity does not modify the domain-invariant feature space even after
the long-time adaptation (i.e., not forgetting source knowledge (a, b)).
Moreover, when applying domain-specific block selection to other methods
instead of updating all BN layers (c, d), we find that our block selection
methods improves other methods as well. In particular, it reduces the error
rate of TENT and EATA by 46.6% and 32.5% in CIFAR10-C, respectively. This can
be interpreted as updating domain-specific blocks can alleviate error
accumulation caused by model collapse, as addressed by [32, 33]. However,
there is no significant improvement in the SAR method. This is because SAR
uses a model reset approach, which restores model parameters to their original
values; thus, the adaptation performance does not differ significantly.

Figure 4: Performance of our framework with various pseudo-label generation
setting including our paired-view (orange) and others. We use ResNext-29A for
CIFAR100-C.

###  5.4 Effect of paired-view consistency

In CoTTA [44], pseudo-labels are generated by averaging the teacher’s
predictions for the given 32 augmented images transformed by random
augmentation including random pad-crop, color jitter, random affine transform,
Gaussian blur, random horizontal flip, and Gaussian noise. In our framework,
we generate pseudo-labels by simply averaging the teacher’s predictions for
two images: original test and horizontally flipped images. This is based on
the insight that the simple flip operation does not create domain gap between
the augmented image and the test image, which can reduces the quality of
pseudo label.

In Figure 4, we compare the adaptation performance of our framework for the
gradual setting with different pseudo-label generation methods. Specifically,
while entropy minimization on the domain-specific blocks is conducted, the
pseudo label is generated by averaging teacher predictions from (i) 2 images
with paired-view (orange; ours), (ii) 32 images with Gaussian noise and
Gaussian blur (blue), (iii) 32 images with color jitter (green), (iv) 32
images with noise, blur, and color jitter (grey), and (v) 32 images with
random pad-crop, affine transform, noise, blur, and color jitter (black;
CoTTA). Note that, random horizontal flip with 0.5 probability is included in
(ii-v). As expected, the adaptation with our pseudo-label generation
outperforms others. It is worth noting that using random pad-crop, which adds
a 16-pixel border to the CIFAR image and subsequently performs random cropping
to a 32x32 size, significantly decreases the pseudo-label quality due to the
potential for objects to be partially cropped.

###  5.5 Single sample test-time adaptation

Since the single prediction for a single input is crucial for some real-time
systems, we consider single-sample TTA as investigated in [8]. In the single-
sample TTA setting, the last $b$ test samples are stored in a memory buffer.
After every $b$ steps, the model parameters are updated by test-time
adaptation methods with a $b$-size batch from the memory. Following [8], we
decrease the learning rate by $\textit{original batch size}/b$ due to the more
frequent updates. In this setting, the challenge of the TTA is that error
accumulation also increases due to the frequent updates. Table 6 provides the
results for single-sample TTA with various buffer sizes $b$. We observed that
previous methods suffer from a small batch-size, but our method is relatively
strong across various buffer sizes. This demonstrates that our method
alleviates error accumulation caused by frequent updates through proper
pseudo-label generation.

Method | Window size | Mean
---|---|---
8 | 16 | 32 | 64
Source only | 43.5 | 43.5
BN-1 | 26.2 | 23.1 | 21.5 | 20.8 | 22.9
TENT | 23.6 | 20.2 | 18.6 | 18.9 | 20.3
CoTTA | 27.4 | 37.5 | 17.1 | 15.0 | 24.3
EATA | 22.9 | 19.2 | 17.6 | 17.9 | 19.4
RMT | 32.3 | 21.8 | 15.8 | 12.4 | 20.6
DPLOT (ours) | 16.4 | 13.1 | 11.6 | 11.3 | 13.1

Table 6: Classification rate (@level 1-5) in the gradual setting of the
CIFAR10-C benchmark with WRN28-10, using single-sample TTA, while considering
different buffer sizes $b$.

##  6 Conclusion

In this work, we propose DPLOT to address proper pseudo-label generation. The
proposed framework is based on two components: domain-specific block selection
before deployment and paired-view pseudo-labeling. After deployment, we use
entropy minimization to update blocks involved in domain-specific feature
extraction. Subsequently, we employ paired-view consistency loss, which forces
the model to produce the exact prediction of the pseudo-label generated by
averaging the teacher’s predictions for the test and its corresponding flipped
inputs. Extensive experiments demonstrated that our framework outperforms
previous competitive methods by a large margin in TTA benchmarks. Also, DPLOT
does not modify the network during the training stage, making it easily
applicable.

## Acknowledgement

This work was partially supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government
(MSIT) (No. 2022-0-00951, Development of Uncertainty Aware Agents Learning by
Asking Questions) and by LG Electronics and was collaboratively conducted with
the Advanced Robotics Laboratory within the CTO Division of the company.

## References

  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Benz et al. [2021] Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon.  Revisiting batch normalization for improving corruption robustness.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 494–503, 2021.
  * Chen et al. [2022] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi.  Contrastive test-time adaptation.  In _CVPR_ , 2022.
  * Choi et al. [2022] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun.  Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes.  In _European Conference on Computer Vision_ , pages 440–458. Springer, 2022.
  * Croce et al. [2020] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein.  Robustbench: a standardized adversarial robustness benchmark.  _arXiv preprint arXiv:2010.09670_ , 2020.
  * Cygert and Czyżewski [2020] Sebastian Cygert and Andrzej Czyżewski.  Toward robust pedestrian detection with data augmentation.  _IEEE Access_ , 8:136674–136683, 2020.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Döbler et al. [2023] Mario Döbler, Robert A Marsden, and Bin Yang.  Robust mean teacher for continual and gradual test-time adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7704–7714, 2023.
  * French et al. [2018] Geoff French, Michal Mackiewicz, and Mark Fisher.  Self-ensembling for visual domain adaptation.  In _International Conference on Learning Representations_ , 2018.
  * Galloway et al. [2019] Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor.  Batch normalization is a cause of adversarial vulnerability.  _arXiv preprint arXiv:1905.02161_ , 2019.
  * Ganin and Lempitsky [2015] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In _International conference on machine learning_ , pages 1180–1189. PMLR, 2015.
  * Geirhos et al. [2018] Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann.  Generalisation in humans and deep neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Grandvalet and Bengio [2004] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  _Advances in neural information processing systems_ , 17, 2004.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _Proceedings of the International Conference on Learning Representations_ , 2019.
  * Hendrycks et al. [2020] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  AugMix: A simple data processing method to improve robustness and uncertainty.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2020.
  * Hendrycks et al. [2021] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _ICCV_ , 2021.
  * Hoffman et al. [2018] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In _International conference on machine learning_ , pages 1989–1998. Pmlr, 2018.
  * Hoyer et al. [2023] Lukas Hoyer, Dengxin Dai, Haoran Wang, and Luc Van Gool.  Mic: Masked image consistency for context-enhanced domain adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11721–11732, 2023.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. pmlr, 2015.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.
  * Kireev et al. [2022] Klim Kireev, Maksym Andriushchenko, and Nicolas Flammarion.  On the effectiveness of adversarial training against common corruptions.  In _Uncertainty in Artificial Intelligence_ , pages 1012–1021. PMLR, 2022.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Krizhevsky et al. [2009] Alex Krizhevsky et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kundu et al. [2020] Jogendra Nath Kundu, Rahul Mysore Venkatesh, Naveen Venkat, Ambareesh Revanur, and R Venkatesh Babu.  Class-incremental domain adaptation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16_ , pages 53–69. Springer, 2020.
  * Li et al. [2017] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou.  Revisiting batch normalization for practical domain adaptation, 2017.
  * Liang et al. [2023] Jian Liang, Ran He, and Tieniu Tan.  A comprehensive survey on test-time adaptation under distribution shifts.  _arXiv preprint arXiv:2303.15361_ , 2023.
  * Lopes et al. [2019] Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk.  Improving robustness without sacrificing accuracy with patch gaussian augmentation.  _arXiv preprint arXiv:1906.02611_ , 2019.
  * Lu et al. [2022] Yulei Lu, Yawei Luo, Antao Pan, Yangjun Mao, and Jun Xiao.  Domain generalization with global sample mixup.  In _European Conference on Computer Vision_ , pages 518–529. Springer, 2022.
  * Marsden et al. [2022] Robert A Marsden, Mario Döbler, and Bin Yang.  Gradual test-time adaptation by self-training and style transfer.  _arXiv preprint arXiv:2208.07736_ , 2022.
  * Mirza et al. [2022] M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof.  The norm must go on: Dynamic unsupervised domain adaptation by normalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14765–14775, 2022.
  * Niu et al. [2022] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan.  Efficient test-time model adaptation without forgetting.  In _International conference on machine learning_ , pages 16888–16905. PMLR, 2022.
  * Niu et al. [2023] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan.  Towards stable test-time adaptation in dynamic wild world.  In _Internetional Conference on Learning Representations_ , 2023.
  * Prasanna et al. [2023] B Prasanna, Sunandini Sanyal, and R Venkatesh Babu.  Continual domain adaptation through pruning-aided domain-specific weight modulation.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 2457–2463. IEEE, 2023.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR, 2019.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In _Advances in Neural Information Processing Systems (NIPS)_ , 2015\.
  * Schneider et al. [2020] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  _Advances in neural information processing systems_ , 33:11539–11551, 2020.
  * Shannon [1948] C. E. Shannon.  A mathematical theory of communication.  _The Bell System Technical Journal_ , 27(3):379–423, 1948.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ , 30, 2017.
  * Vu et al. [2019] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2517–2526, 2019.
  * Wang et al. [2021a] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In _International Conference on Learning Representations_ , 2021a.
  * Wang et al. [2022] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai.  Continual test-time domain adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7201–7211, 2022.
  * Wang and Qi [2022] Xiao Wang and Guo-Jun Qi.  Contrastive learning with stronger augmentations.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(5):5549–5560, 2022.
  * Wang et al. [2019] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 322–330, 2019.
  * Wang et al. [2021b] Yu Wang, Rui Zhang, Shuo Zhang, Miao Li, YangYang Xia, XiShan Zhang, and ShaoLi Liu.  Domain-specific suppression for adaptive object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9603–9612, 2021b.
  * Wu et al. [2021a] Aming Wu, Yahong Han, Linchao Zhu, and Yi Yang.  Instance-invariant domain adaptive object detection via progressive disentanglement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(8):4178–4193, 2021a.
  * Wu et al. [2021b] Aming Wu, Rui Liu, Yahong Han, Linchao Zhu, and Yi Yang.  Vector-decomposed disentanglement for domain-invariant object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9342–9351, 2021b.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In _BMVC_ , 2016.
  * Zhao et al. [2023] Bowen Zhao, Chen Chen, and Shu-Tao Xia.  DELTA: DEGRADATION-FREE FULLY TEST-TIME ADAPTATION.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Zhou et al. [2021] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain generalization with mixstyle.  In _ICLR_ , 2021.
  * Zhou et al. [2022] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.


# The folding of art: avoiding one’s past in finite space

Anders Levermann<EMAIL_ADDRESS>http://www.pik-
potsdam.de/~anders Department of Complexity Science, Potsdam Institute for
Climate Impact Research, Potsdam, Germany  Lamont-Doherty Earth Observatory,
Columbia University, Palisades, NY, USA.  Department of Physics, Potsdam
University, Potsdam, Germany.

###### Abstract

Through-out human history the new generations have sought to create their own
artistic style while trying to avoid repeating, for example, earlier
generations’ music. If we assume that this search occurs in a multi-
dimensional but confined space of creativity, this gives rise to highly
complex dynamics. We present a very simple mathematical model with two
parameters which can serve as a generic representation for past avoidance in
finite space and is qualitatively distinct from earlier dynamical systems. In
the presented radially confined form, the trajectory preserves its complexity
while retracting to the vicinity of a hypersurface of constant radius when
considered in higher dimension.

Dynamical systems, Complexity, Non-linear dynamics, SOC, Chaos

††preprint: APS/123-QED

Complex dynamical behaviour that emerges from simple rules of a small number
of actors is still a non-trivial phenomenon despite it being the subject of
non-linear dynamical systems theory for decades. Complex but structured
behaviour can emerge from deterministic non-linear dynamics of a single
trajectory forming, for example, a strange attractor over time [1, 2] or from
spatially evolving patterns such as Laplacian growth [3, 4, 5] or it can
emerge from unstructured stochasticity such as, for example, in the time
evolution of single random walkers in diffusion-limited aggregation [6, 7, 8]
or simultaneously evolving fracture dynamics [9, 10, 11] or self-organized
criticality [12, 13, 14, 15]. There are many qualitatively different ways in
which complexity can arise from simple rules.

Here we present a deterministic motion in $D$-dimensional space which shows
highly complex behaviour through avoiding its own past and being confined to a
finite space. The complexity arises from an ever expanding dynamics that is
confined to finite space. The main difference to traditional dynamical systems
is that it is non-local in time. We present two mathematical representations:
one through a time-dependent potential landscape and the other through an
integro-differential equation. Geometrically the systems exhibits the
trajectory folding that is found in strange attractors and is thereby a
relatively simple and pure prototype of the dynamical folding principle. Note
however that the trajectory can cross itself because it tracks its history and
thereby the state space is not a phase space.

As a motivation for the model we suggest the path of society in search for
cultural innovation. It can also be considered a paradigmatic system for
infinite economical growth on a finite planet with finite resources. The
equations themselves resemble those of a relativistic description of a charged
particle via the Liénard-Wiechert potential [16, 17], in which the electron is
repelled by its own trajectory. But these are merely illustrations of possible
applications. The main result is the complexity that arises from a spatial
confinement of infinite growth.

In order to capture, for example, the path of society in an abstract
creativity space we consider its trajectory ${\bf r}\in\mathbb{R}^{D}$
changing position along the negative gradient of a landscape $L$.

$\frac{d{\bf r}}{dt}=-{\bf\nabla}L\left({\bf r}\right).$ (1)

The landscape $L$ is defined everywhere within a certain bounded domain $\cal
B$ as the sum

$L\left({\bf x},t\right)=L_{b}\left({\bf x}\right)+L_{h}\left({\bf
x},t\right).$ (2)

of a boundary potential, $L_{b}$, and a history potential, $L_{h}$, with ${\bf
x}\in{\cal B}\subset\mathbb{R}^{D}$. Here we choose the bounded domain to be
the unit sphere in $D$ dimensions. The time-independent boundary potential,
$L_{b}$, is thus defined as

$L_{b}\left({\bf x}\right)=\left(1-\left|{\bf x}\right|\right)^{-1}.$ (3)

with $|{\bf x}|\equiv\sqrt{\sum_{j=1}^{D}x_{j}^{2}}$ the absolute value. The
history potential, $L_{h}$, is defined through an ordinary differential
equation via

$\frac{dL_{h}}{dt}\left({\bf x}\right)=\left(1+\frac{\left|{\bf r}(t)-{\bf
x}\right|}{\rho}\right)^{-1}-\frac{L_{h}}{\tau}.$ (4)

with positive constants $\rho>0$ and $\tau>0$. These two parameters have a
clear interpretation. While $\tau$ controls the memory of society with respect
to its history, $\rho$ defines the space in creativity space that is taken by
the current generation. The assumption that these are constant is made for
simplicity.

Figure 1: Generic trajectory of the dynamics (panel a) in $D=2$ with
parameters $\rho=0.1$ and $\tau=50$ and (panel b) in $D=3$ dimensions with
$\rho=0.1$ and $\tau=100$. Shown is the initial time evolution up to $T=200$
for initial con

The kernel of the history potential (fig. S1 in the supplementary material)

$K\left({\bf r}(t),{\bf x}\right)=\left(1+\frac{\left|{\bf r}(t)-{\bf
x}\right|}{\rho}\right)^{-1}=\frac{\rho}{\rho+\left|{\bf r}(t)-{\bf
x}\right|}$ (5)

is one at the current position of the trajectory $K\left({\bf r}(t),{\bf
x}={\bf r}(t)\right)=1$, and declines (quasi-hyperbolically) to $K=0.5$ at a
distance $\rho$ from the current trajectory position ${\bf r}(t)$. Thus
according to equation (4) the trajectory generates an exponentially decaying
tail along its path. Thus fittingly, the set of equations can be rewritten as
an integro-differential equation (see appendixB).

If we start with a generic history landscape that is equal to zero everywhere
$L_{h}({\bf x},t=0)=0;\forall\;{\bf x}\in\mathbb{R}^{D}$, then the trajectory
${\bf r}(t)$ is fully determined by its initial position ${\bf r}(t=0)={\bf
r_{0}}$ which, in principle, is an additional free parameter, but the
statistics of the emerging attractor does not seem to be dependent on this
initial condition. A coefficient that weighs the two contributions from
$L_{b}$ and $L_{h}$ can be incorporated into the choice of the radius of the
sphere of the boundary potential. Here we chose the unit circle and an equal
weighting of the two potentials for illustrative purposes.

Starting from any initial condition the resulting trajectories are complex and
form a ball in their respective creativity space (fig. 1 and animation in
supplementary information). Without proof we hypothesis that the statistics of
the emerging trajectory is independent of the initial condition. There might
be a specific set of initial conditions that yield regular behaviour, but
placing the initial condition in the origin does yield the same complex
behaviour that cannot be easily distinguished from that of fig.1. The
trajectory exhibits strong sensitivity to the initial condition (Fig. S2), but
the long-term trajectory density does not. It only depends on the two
parameters, i.e. the systems memory $\tau$ and the space occupied by the
current generation $\rho$.

Due to the exponential decay of the memory the long-term average of the
history potential is likely to reach a steady state given by

$\left<L_{h}\right>(r)=\frac{\tau\rho}{2\pi}\int_{0}^{1}\int_{0}^{2\pi}\frac{p(r^{\prime}){r^{\prime}}^{D-1}dr^{\prime}d\theta}{\rho+\sqrt{r^{2}+{r^{\prime}}^{2}-2rr^{\prime}\cos(\theta)}}$
(6)

where $p(r)$ is the long-term trajectory density which is given by an implicit
equation

$\frac{\tau\rho}{2\pi}\int_{0}^{1}\int_{0}^{2\pi}\frac{\partial}{\partial
r}\frac{p(r^{\prime}){r^{\prime}}^{D-1}dr^{\prime}d\theta}{\rho+\sqrt{r^{2}+{r^{\prime}}^{2}-2rr^{\prime}\cos(\theta)}}=\frac{1}{(1-r)^{2}}.$
(7)

as derived in the appendix.

When considered in higher dimensions $D$, the dynamics retracts onto a thin
shell around a hypersphere of a constant radius and some trajectory density
near the origin (fig. 2). Fig. 3 shows the trajectory points that pass through
a two-dimensional plane ${\cal P}=\\{x\in{\cal R}^{D}|x_{1}=x_{2}=0\\}$. This
retraction towards the vicinity of a hypersurface does, however, not reflect a
regularization of the dynamics. Also in higher dimensions the dynamics remains
irregular while being fully deterministic. As an example fig. 4 shows the time
series of the five components of the Cartesian coordinates of the dynamics in
$D=5$ dimensions after an arbitrarily chosen time of $t=1800$.

The deterministic system exhibits complex behaviour and even though the long-
term radial trajectory distribution can be computed through a relatively
simple equation (7), this does not capture the complexity of the time
evolution. We hypothesize that the trajectory lies densely within the
spherical shell it inhabits and is thereby ergodic. The simple set of
equations (1-4) constitutes a paradigmatic dynamics of past avoidance in
finite space and a deterministic complexity dynamics that is qualitatively
distinct from common deterministic chaos.

Figure 2: Density of trajectory as a function of radius $r$ in (panel a)
$D=2,3,4$ dimensions and (panel b) $D=2,3,4,5,6,7$ dimensions for parameters
$\rho=0.1$ and $\tau=10$ which yields the convergence radius estimate of
$r_{m}\approx 1/3$. $p(r)$ scales with $r^{D}$ such that
$\int_{0}^{1}dr\;r^{D}p(r)=1$ Figure 3: Projection of trajectory cloud onto
plane $\cal P$ through origin in different dimensions $D$. Figure 4: Example
of a part of the time series of all components in $D=5$.

##  Appendix A Appendix

##  Appendix B Model in integro-differential form

The model (Eq. 1-4) can be reorganized into an integro-differential equation.
Since the boundary potential is time-independent its gradient at the position
${\bf r}(t)$ is simply

${\bf\nabla}L_{b}\left({\bf r}\right)=\left(1-r(t)\right)^{-2}\cdot\frac{{\bf
r(t)}}{r(t)}.$ (8)

with $r(t)=|{\bf r(t)}|$. Time integration and spatial differentiation of the
history potential yields

${\bf\nabla}L_{h}\left({\bf
r}(t),t\right)=\int_{0}^{t}dt^{\prime}\frac{\rho\cdot
e^{-\frac{t-t^{\prime}}{\tau}}}{\left(\rho+\left|{\bf r}(t)-{\bf
r}(t^{\prime})\right|\right)^{2}}\cdot\frac{{\bf r}(t)-{\bf
r}(t^{\prime})}{\left|{\bf r}(t)-{\bf r}(t^{\prime})\right|}.$ (9)

which results into the integro-differential form

$\displaystyle\frac{d{\bf r}}{dt}(t)$ $\displaystyle=$
$\displaystyle-\left(1-r(t)\right)^{-2}\cdot\frac{{\bf r(t)}}{r(t)}$ (10)
$\displaystyle-\int_{0}^{t}dt^{\prime}\frac{\rho\cdot
e^{-\frac{t-t^{\prime}}{\tau}}}{\left(\rho+\left|{\bf r}(t)-{\bf
r}(t^{\prime})\right|\right)^{2}}\cdot\frac{{\bf r}(t)-{\bf
r}(t^{\prime})}{\left|{\bf r}(t)-{\bf r}(t^{\prime})\right|}.$

##  Appendix C Emergent steady state potential

When taking the limit of the time-average of equ. (4) the left hand side
yields

$\lim_{T\rightarrow\infty}\frac{1}{T}\int_{0}^{T}\frac{dL_{h}}{dt}\left({\bf
x}\right)=\lim_{T\rightarrow\infty}\frac{1}{T}\left(L_{h}(T)-L_{h}(0)\right)=0$
(11)

as long as the potential remains finite within $\cal B$. The right hand side
becomes

$\displaystyle\lim_{T\rightarrow\infty}\frac{1}{T}\int_{0}^{T}\left(1+\frac{\left|{\bf
r}(t)-{\bf
x}\right|}{\rho}\right)^{-1}-\lim_{T\rightarrow\infty}\frac{1}{T}\int_{0}^{T}\frac{L_{h}}{\tau}$
$\displaystyle=\lim_{T\rightarrow\infty}\frac{1}{T}\int_{0}^{T}\left(1+\frac{\left|{\bf
r}(t)-{\bf x}\right|}{\rho}\right)^{-1}-\frac{\left<L_{h}\right>}{\tau}$

and thus the long-term time average of the history potential, assuming it
exists, can be computed as

$\left<L_{h}\right>=\lim_{T\rightarrow\infty}\frac{\tau}{T}\int_{0}^{T}\left(1+\frac{\left|{\bf
r}(t)-{\bf x}\right|}{\rho}\right)^{-1}.$ (12)

Assuming the emergent potential is radially symmetric and corresponds to a
time-independent probability for the trajectory position $p({\bf r})=p(r)$ the
long-term average of the history potential can be written as

$\displaystyle\left<L_{h}\right>(r)=\tau\cdot\int_{\cal B}d{\bf
r^{\prime}}\frac{\rho\cdot p({\bf r^{\prime}})}{\rho+\left|{\bf r}(t)-{\bf
r^{\prime}}\right|}$ $\displaystyle=$ $\displaystyle\frac{\tau}{2\pi
S_{D-2}}\int_{0}^{1}\int_{0}^{2\pi}\int_{S_{D-2}}\frac{\rho\cdot
p(r^{\prime}){r^{\prime}}^{D-1}dr^{\prime}d\theta
d\Omega}{\rho+\sqrt{r^{2}+{r^{\prime}}^{2}-2rr^{\prime}\cos(\theta)}}$
$\displaystyle=$
$\displaystyle\frac{\tau\rho}{2\pi}\int_{0}^{1}\int_{0}^{2\pi}\frac{p(r^{\prime}){r^{\prime}}^{D-1}dr^{\prime}d\theta}{\rho+\sqrt{r^{2}+{r^{\prime}}^{2}-2rr^{\prime}\cos(\theta)}}$

where $S_{D-2}$ is the surface of $D-2$ sphere. Following a similar reasoning
as above the long-term time average of the left hand side of equ. (1) vanishes

$\lim_{T\rightarrow\infty}\frac{1}{T}\int_{0}^{T}\frac{d{\bf
r}}{dt}=\lim_{T\rightarrow\infty}\frac{1}{T}\left({\bf r}(T)-{\bf
r}(0)\right)={\bf 0}$ (13)

which yields an implicit equation for the probability density function $p(r)$

$\frac{\tau\rho}{2\pi}\int_{0}^{1}\int_{0}^{2\pi}\frac{\partial}{\partial
r}\frac{p(r^{\prime}){r^{\prime}}^{D-1}dr^{\prime}d\theta}{\rho+\sqrt{r^{2}+{r^{\prime}}^{2}-2rr^{\prime}\cos(\theta)}}=\frac{1}{(1-r)^{2}}.$
(14)

## References

  * Lorenz [1963] E. N. Lorenz, Deterministic Nonperiodic Flow, Journal of the Atmospheric Sciences 20, 130 (1963).
  * Sprott [2014] J. C. Sprott, A dynamical system with a strange attractor and invariant tori, Physics Letters, Section A: General, Atomic and Solid State Physics 378, 1361 (2014).
  * Paterson [1981] L. Paterson, Radial fingering in a hele shaw cell, Journal of Fluid Mechanics 113, 513–529 (1981).
  * Barra  _et al._ [2001] F. Barra, B. Davidovitch, A. Levermann, and I. Procaccia, Laplacian Growth and Diffusion Limited Aggregation: Different Universality Classes, Physical Review Letters 87, 134501 (2001), 0103126 [cond-mat] .
  * Hentschel  _et al._ [2002] H. G. E. Hentschel, A. Levermann, and I. Procaccia, Transition in the fractal properties from diffusion-limited aggregation to Laplacian growth via their generalization, Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics 66, 1 (2002).
  * Witten and Sanders [1981] T. Witten and L. Sanders, Diffusion-Limited Aggregation, a Kinetic Critical Phenomenon, Physical Review Letters 47, 1400 (1981).
  * Hastings and Levitov [1998] M. Hastings and L. Levitov, Laplacian growth as one-dimensional turbulence, Physica D 116, 244 (1998).
  * Davidovitch  _et al._ [2001] B. Davidovitch, M. H. Jensen, A. Levermann, J. Mathiesen, and I. Procaccia, Thermodynamic formalism of the harmonic measure of diffusion limited aggregates: phase transition, Phys Rev Lett 87, 164101 (2001), 0107024 [cond-mat] .
  * Lemaire  _et al._ [1991] E. Lemaire, P. Levitz, G. Daccord, and H. van Damme, From Viscouls Fingering to Viscoelastic Fracturing in Colloidal Fluids, Physical Review Letters 67, 2009 (1991).
  * Zhao and Maher [1993] H. Zhao and J. Maher, Associating-polymer effect in a Hele-Shaw experiment, Physical Review E 47, 4278 (1993).
  * Levermann and Procaccia [2002] A. Levermann and I. Procaccia, Bi-Laplacian Growth Patterns in Disordered Media, Physical Review Letters 89, 2 (2002).
  * Bak  _et al._ [1987] P. Bak, C. Tang, and K. Wiesenfeld, Self-organized criticality: An explanation of the 1/f noise, Physical Review Letters 59, 381 (1987).
  * Tang and Bak [1988] C. Tang and P. Bak, Mean field theory of self-organized critical phenomena, Journal of Statistical Physics 51, 797 (1988).
  * Peters  _et al._ [2002] O. Peters, C. Hertlein, and K. Christensen, A complexity view of rainfall., Physical review letters 88, 018701 (2002), 0201468 [cond-mat] .
  * Scheinkman and Woodford [1994] J. Scheinkman and M. Woodford, Self-organized criticality and economic fluctuations, American Economic Review 84, 417 (1994).
  * Liénard [1898] A.-M. Liénard, (1898).
  * Wiechert [1901] E. Wiechert, Elektrodynamische Elementargesetze, Annalen der Physik 309, 667 (1901).