# Unsupervised Adaptation of Semantic Segmentation Models without Source Data

Sujoy Paul, Ansh Khurana, Gaurav Aggarwal
Google Research
{sujoyp, anshkhurana<EMAIL_ADDRESS>
###### Abstract

We consider the novel problem of unsupervised domain adaptation of source
models, without access to the source data for semantic segmentation.
Unsupervised domain adaptation aims to adapt a model learned on the labeled
source data, to a new unlabeled target dataset. Existing methods assume that
the source data is available along with the target data during adaptation.
However, in practical scenarios, we may only have access to the source model
and the unlabeled target data, but not the labeled source, due to reasons such
as privacy, storage, etc. In this work, we propose a self-training approach to
extract the knowledge from the source model. To compensate for the
distribution shift from source to target, we first update only the
normalization parameters of the network with the unlabeled target data. Then
we employ confidence-filtered pseudo labeling and enforce consistencies
against certain transformations. Despite being very simple and intuitive, our
framework is able to achieve significant performance gains compared to
directly applying the source model on the target data as reflected in our
extensive experiments and ablation studies. In fact, the performance is just a
few points away from the recent state-of-the-art methods which use source data
for adaptation. We further demonstrate the generalisability of the proposed
approach for fully test-time adaptation setting, where we do not need any
target training data and adapt only during test-time.

##  1 Introduction

Figure 1: Overview. Unsupervised domain adaptation methods for semantic
segmentation in literature use labeled source dataset while adapting to an
unlabeled target dataset. However, in this work, we only use the knowledge
from the source model for adaptation without access to the source data.

Deep neural networks have shown immense success for dense prediction tasks,
such as semantic segmentation [5]. However, models learned on one dataset, say
the source, may not generalize well to a new dataset, say the target.
Annotating every new target dataset is expensive, especially for the semantic
segmentation task. For example, a single image from the Cityscapes dataset
required about 1.5 hours to annotate [9]. To avoid the requirement of
annotated target data, Unsupervised Domain Adaptation (UDA) methods [52, 15]
adapt the model learned with labeled source data to the target dataset,
without needing annotated target images.

In the literature, UDA methods for semantic segmentation assume that data
points from both the source and the target datasets are available for
adaptation. However, such an assumption may not be practical, where either the
source dataset cannot be shared to maintain privacy or the source dataset may
take up a lot of storage space. In this case, one has access to only the
light-weight source model which was trained using labeled source data. This
would restrict sharing private information as well as require much less
storage. For example, storing the GTA5 dataset [40] used as source in most UDA
methods, needs $57$ GB of storage whereas a model trained using the same data
requires only $0.17$ GB. With this motivation, in this work, we do not assume
access to any source data and adapt the source model using only the unlabeled
target data for semantic segmentation. Figure 1 illustrates the problem
pictorially.

Before diving into the details of source-free adaptation, let’s study the
techniques used in UDA for semantic segmentation using source data. They can
be categorized mainly into three groups: 1) output space adaptation [50, 16]
that aligns the source and target output distributions, 2) pixel space
adaptation [15, 8] that translates the source image distribution to the target
distribution, and uses them for training, and finally, 3) pseudo labeling [66,
26] that predicts the labels of the target images using the source model and
uses them for training. When we do not have access to the source data, the
first two methods are not even applicable. Recent methods on UDA for
classification task [27, 60] without source data largely follow the third
strategy, i.e., pseudo-labeling. A few methods in this line of work also
generate data from the target distribution to boost the performance [23]. Very
recently published works [28, 42] for source-free adaptation of segmentation
models are no different. Additionally, [28] also follow the first option by
dividing the target data into easy (source) and hard (target) group and train
an additional discriminator.

In this work, we propose a novel consistency based self-training approach to
adapt the source model using unlabeled target images. As the target data comes
from a shifted distribution than the source, we first update the norm
parameters of the network using the unlabeled target data. Then, we extract
knowledge from the source model by pseudo-labeling the unlabeled target. The
extracted pseudo-labels should be consistent against certain spatial
transformations on images such as rotation, flipping, mirroring, and image
augmentations such as cutout, Gaussian blur, etc. We use these consistencies
to enhance the knowledge extracted from the source model. This can be posed as
a constrained optimization problem, where in each iteration, we first compute
the gradients of the target model using the actual pseudo-labels with respect
to the source model, and then obtain the gradients using the consistency
losses computed with respect to the current target model. We show via
empirical analysis that these simple consistencies with pseudo-labeling help
to considerably improve the performance of the model on the target images. We
also create a pseudo image distribution by collaging pairs of images, which
enhances the variety of images in the dataset, and helps to regularize the
model.

We evaluate our framework on three different settings: GTA5 $\rightarrow$
Cityscapes and SYNTHIA $\rightarrow$ Cityscapes for outdoor scenes and the
novel setting of SceneNet $\rightarrow$ SUN for indoor scenes, showcasing the
generic nature of our algorithm. There is no known effort in the literature to
show domain adaptation results on the mentioned indoor setting. Further, to
demonstrate the generalisability of our consistency based approach, we apply
it in the fully test-time adaptation setting [55], where it outperforms loss
functions proposed by recent methods dedicated for fully test time adaptation
[35, 55] on all the above three dataset settings. The main contributions of
our work are:

  * •

We tackle the interesting problem of source-free adaptation for semantic
segmentation and propose a self-training approach by imposing certain input-
output consistencies to extract the knowledge from the source model.

  * •

Empirical analysis shows that our method establishes the new state-of-the-art
for source-free adaptation, and is only a few points away from the state-of-
the-art which uses the entire labeled source dataset to adapt.

  * •

Our method works well even for test time adaptation, where we do not access
even the unlabeled target, and the adaptation is done individually for each
test sample.

##  2 Related Works

Figure 2: Framework overview. (Left) Given the source model, we first update
the norm parameters of the network by just forward propagating the unlabeled
target images, to obtain the model $\boldsymbol{\mathrm{M}}_{s}$. (Right) Then
we learn a new target model $\boldsymbol{\mathrm{M}}_{t}$ using self-training.
We first obtain the pseudo-labels using $\boldsymbol{\mathrm{M}}_{s}$ and
compute the classification loss $\mathcal{L}_{c}$. Then, we transform the
input image ($\boldsymbol{\mathrm{T}}_{i}$) as well as the target model’s
prediction ($\boldsymbol{\mathrm{T}}_{o}$) in the current iteration and impose
certain consistencies via the loss function $\mathcal{L}_{r}$. On the right-
hand side, each arrow color denotes one full pass, either forward or
forward+backward propagation.

We divide the related works discussion in two parts: 1) methods using the
source data to adapt (most works in literature), and 2) methods that do not
use any source data, (a few recent efforts, mostly for classification and a
couple for the segmentation task).

UDA with Source Data.

UDA methods for the image classification task aim to align the source and
target distributions. To do so, maximum mean discrepancy [29] and adversarial
learning [13, 52] based approaches have been proposed. A few algorithms focus
on improving deep models [30, 43, 21, 10] and translating the input source
images to the target domain [2, 15, 49]. For the more challenging task of
semantic segmentation, existing UDA methods can be categorized primarily into
three groups: output alignment, pixel-adaptation and pseudo-labeling. In
output alignment, the methods aim at aligning the output or feature
distribution between the source and target domains[50, 6, 7, 16, 64, 1]. For
pixel alignment, similar to as in image classification, the algorithms
translate the input images to the target domain by changing style, while
preserving content information [4, 8, 15, 36, 57, 65, 59]. For pseudo-
labeling, methods aim to generate pixel-wise pseudo labels on the target
images, which is utilized to fine-tune the segmentation model trained on the
source [44, 66, 26, 63, 22, 38, 45, 34, 11]. There are also several methods in
literature that try to combine these strategies [12, 25, 51, 54, 39, 56, 37,
19, 31, 17, 47]. Compared to these approaches, we assume no access to the
source dataset, which is a more realistic setting but makes the task much more
challenging.

UDA without Source Data.

Unlike the above methods, there have been a few works for classification tasks
which do not use source data, but only the source model for adaptation. The
methods involve - entropy minimization with divergence maximization [27],
pseudo-labeling with self-reconstruction [60], generating additional target
images [23] and self-supervision [58]. For source-free adaptation for semantic
segmentation, recently, [28] proposed an algorithm that combines ideas from
the above methods like image generation, pseudo-labeling and output space
adaptation by dividing the target into easy and hard, and learning an
additional discriminator to align them. In [42], pseudeo-labeling and
robustness to dropout randomness is used to tackle the problem, which performs
very similar to our strong baseline of Pseudo-Label (PL). Pseudo-Labeling is
also used in [61] with the concept of negative learning, i.e., the pixels
where the model is not confident towards a certain positive label, it may be
confident towards multiple negative labels. Very recently, a method for robust
training of source models is proposed in [20], followed by pseudo-labeling on
the target. In comparison to this method, we do not assume access to the
source data or training process but only the trained source model.

##  3 Domain Adaptation from Source Models

Figure 3: Transforms. This figure shows the different transforms used in our
algorithm. The left-most side shows an image with the pseudo-labels obtained
from the source model $\boldsymbol{\mathrm{M}}_{s}$, which are used to compute
the loss $\mathcal{L}_{c}$. The top row shows the transformations
individually, followed by their random compositions in the second row, and the
last row shows the transformations on the output of the target model for a
certain iteration, which are used a ground-truth for computing loss
$\mathcal{L}_{r}$.

In this section, we formally define the problem statement, followed by an
overview of our framework, and finally a detailed description of each blocks
in our framework.

###  3.1 Problem Statement

Consider that we have a source model for semantic segmentation, which needs to
be adapted to a new target domain, given only unlabeled target images, without
access to the source images on which the source model was trained. Formally,
consider we have a source model
$\boldsymbol{\mathrm{M_{s}}}:\mathbb{R}^{H\times W\times
3}\rightarrow\mathbb{R}^{H\times W\times C}$, which takes as input an RGB
image and predicts for every pixel its category. $H,W$ are the height and
width of the image and $C$ is the number of categories/labels. Now, given an
unlabeled target dataset, $\mathcal{D}_{t}=\\{X_{i}\\}_{i=1}^{n}$, our goal is
to adapt the source model $\boldsymbol{\mathrm{M_{s}}}$ to a new model
$\boldsymbol{\mathrm{M_{t}}}$, such that it performs better on images drawn
from the target distribution, than directly using the source model on the
target images.

###  3.2 Overall Framework

An overview of our proposed framework is presented in Figure 2. As the target
images come from a different distribution than what the source model is
trained with, we first update only the norm parameters of the network, i.e.,
mean and variance of BatchNorm or InstanceNorm, by just forward propagating
the target domain images $\mathcal{D}_{t}$ through the given source model.
This does not change the network weights, but only the norm parameters are
updated. After that, we use the updated source model
$\boldsymbol{\mathrm{M}}_{s}$ to pseudo-label the target images only for the
confident pixels. Along with these pseudo-labels, we impose two different
types of consistency constraints using augment and spatial transformations.
This helps to efficiently extract knowledge from the source model that in turn
helps the adapted model perform better on the target images. Additionally, we
create a pseudo image distribution by collaging pairs of images, thus creating
a wider variety of image compared to the target dataset. We learn a new target
model $\boldsymbol{\mathrm{M}}_{t}$ by simultaneously optimizing for the
pseudo-labels with the consistency constraints.

###  3.3 Update Normalization Parameters

BatchNorm [18] and InstanceNorm [53] are commonly used in deep neural networks
to avoid over-fitting, stabilize training, and faster convergence. For every
activation layer in the network, these normalization methods estimate two
parameters during training, viz., the mean and variance, which are then used
in the testing phase. However, these parameters are only a good estimate for
the images in the training distribution. Thus, in domain adaptation, where the
target data may belong to a different distribution than the source, it may be
useful to first update the mean and variance of the normalization layers,
using the unlabeled target data. This is inspired by works in literature [24,
3] where they estimate domain-specific normalization parameters, while having
access to both the source and the target data, during training.

In order to update the normalization statistics, we only forward propagate the
unlabeled target images through the network for one epoch and use the
following update rule for a particular layer in each iteration.

$\displaystyle\mu$
$\displaystyle\leftarrow(1-\nicefrac{{1}}{{i}})\mu+\frac{1}{i}\mu_{i}$ (1)
$\displaystyle\sigma^{2}$
$\displaystyle\leftarrow(1-\nicefrac{{1}}{{i}})\sigma^{2}+\frac{1}{i}\sigma^{2}_{i}$
(2)

where $i=\\{1,\dots,n\\}$. $\mu_{i}$ and $\sigma^{2}_{i}$ are the mean and
variance of that layer with $X_{i}$ as input. Note that we use instance
normalization in all our experiments. This is because in semantic
segmentation, to handle high resolution images, the batch size can typically
only be one. Further, to estimate the mean and variance on the new target
data, we need to forward propagate each target image only once, without any
backward propagation. This step only updates the normalization parameters,
mean and variance, and not the weights of the network. We call this updated
model $\boldsymbol{\mathrm{M}}_{s}$, which we use to learn the target model
$\boldsymbol{\mathrm{M}}_{t}$, as discussed next.

###  3.4 Pseudo-Labeling with Constraints

Given the normalization updated source model $\boldsymbol{\mathrm{M}}_{s}$, we
use it to label the target images via confidence-filtered pseudo labeling.
Additionally, we impose consistency using certain transforms, which gives the
model an opportunity to improve beyond learning from just the pseudo-labeled
dataset.

####  3.4.1 Pseudo-Labeling:

The pseudo-labeling function
$\hat{Y}=\boldsymbol{\mathrm{PL}}(\boldsymbol{\mathrm{M}}(X))$ to label an
image $X$ with a model $\boldsymbol{\mathrm{M}}$ can be defined as follows:

$\hat{Y}^{x,y}=\begin{cases}j^{*}=\operatorname*{arg\,max}_{j}[\boldsymbol{\mathrm{M}}(X)]^{x,y},&{\scriptstyle\text{if}\max_{j}[\boldsymbol{\mathrm{M}}(X)]^{x,y}>p_{j^{*}}}\\\
\text{No Label},&\text{otherwise}\end{cases}$ (3)

where the superscript $x,y$, represents the spatial co-ordinates,
$j\in\\{1,\dots,C\\}$, and $[\boldsymbol{\mathrm{M}}(X)]^{x,y}$ is a
$C$-dimensional vector summing to 1, representing the pixel’s prediction.
$p_{j}$ is the threshold for prediction confidence of label $j$, which will be
discussed in more detail subsequently. We label the unlabeled target dataset
$\mathcal{D}_{t}$ using the pseudo-labeling function to obtain a new dataset
$\hat{\mathcal{D}}_{t}=\\{X_{i},\hat{Y}_{i}\\}_{i=1}^{n}$, where
$\hat{Y}_{i}=\boldsymbol{\mathrm{PL}}(\boldsymbol{\mathrm{M}}_{s}(X_{i}))$.

We can train a new model $\boldsymbol{\mathrm{M}}_{t}$ using the pseudo-
labeled set $\hat{\mathcal{D}}_{t}$, which would reduce the uncertainties in
the source model $\boldsymbol{\mathrm{M}}_{s}$, offering a better hypothesis
on the target. However, the model can only be as good as the pseudo-labels,
without much room for improvement. We observe that certain transforms on the
target input actually lead to undesirable changes in the output. Thus, we
constrain these to desirable outputs, which acts as a signal to improve beyond
the pseudo-labeled set.

####  3.4.2 Consistency as Constraint

The target network learned using the pseudo labels from Eqn. 3 can only be as
good as the pseudo-labels and it may overfit to that particular set
$\hat{\mathcal{D}}_{t}$. Because of this, we observe that under certain
transformations to the input image such as cutout, Gaussian blur, image
rotation, image mirroring, etc., the network does not predict consistent
labels. By consistent, we mean labels similar to what one would obtain without
applying these transformations. This observation motivates us to use this
consistency as constraint while learning from the pseudo-labeled dataset
$\hat{\mathcal{D}}_{t}$, which acts as an additional signal to improve beyond
pseudo-labeling. We use two different types of image transformations namely -
augment and spatial transforms, as discussed next. Moreover, we create
additional images (collages) by joining pairs of images and apply similar
consistency constraints to further enhance the model.

Augment Transforms $(\mathcal{T}_{a})$:

Cutout and Gaussian blur belong to this transform, as shown in Figure 3. In
this case, we constrain the output to be similar even after applying these
transforms, i.e., if we consider a transform
$\boldsymbol{\mathrm{T}}\in\mathcal{T}_{a}$, then the consistency constraint
can be expressed as

$\boldsymbol{\mathrm{M}}_{t}(\boldsymbol{\mathrm{T}}(X))\approx\boldsymbol{\mathrm{M}}_{t}(X)$
(4)

In cutout, we randomly choose $k$ blocks of size $b\times b$ from the image,
and set them to $0$, where $\frac{kb^{2}}{HW}\approx p$, with two free
parameters $b,p$. $H$ and $W$ are the image height and width. For Gaussian
blur, we choose a kernel size and filter variance. Please refer to the
appendix for details on the parameter values used for these transformations
(Figure 6).

Spatial Transforms $(\mathcal{T}_{s})$:

Image mirroring and rotation fall under this transform set as shown in Figure
3. Compared to augmentations, in this case, the output should not be the same
as obtained from the original image, but should also be transformed. Formally,
considering a transformation $\boldsymbol{\mathrm{T}}\in\mathcal{T}_{s}$, the
consistency constraint can be expressed as

$\boldsymbol{\mathrm{M}}_{t}(\boldsymbol{\mathrm{T}}(X))\approx\boldsymbol{\mathrm{T}}(\boldsymbol{\mathrm{M}}_{t}(X))$
(5)

Note that the order of the model and transform is switched in this case. For
image mirroring, we first randomly choose a column, vertically splitting the
image, and mirror the larger side of the image onto the smaller side. For
rotation, we randomly choose the rotation degree between
$[-5^{\circ},+5^{\circ}]$.

Learning from Image Collages.

Mixup [62] uses weighted combination of image pairs and their labels to
regularize model training. Motivated by this, we propose to combine two images
into a single image, but in the spatial domain, as shown in Figure 4. Given a
pair of images $X_{i},X_{j}$, the collage image is constructed by
concatenating the first half of $X_{i}$ with the second half of $X_{j}$ along
the width. Now, to obtain the pseudo-labels for the collage image, we also
concatenate the pseudo-labels
$\boldsymbol{\mathrm{PL}}(\boldsymbol{\mathrm{M}}_{s}(X_{i}))$ and
$\boldsymbol{\mathrm{PL}}(\boldsymbol{\mathrm{M}}_{s}(X_{j}))$, as shown in
Figure 4. Note that in our algorithm, we learn only from these collage images,
and $X$ from here onwards denotes these collage images. This creates a wider
variety of images that helps to improve the target model.

Figure 4: Process of image and pseudo-label collage creation. We use such
collages to learn the target model.

Optimization Problem:

Using these consistency constraints, we solve the following optimization
problem:

$\displaystyle\min_{\boldsymbol{\mathrm{M}_{t}}}$
$\displaystyle\quad\mathcal{L}_{c}=\sum_{X\in\mathcal{D}_{t}}l_{c}\Big{(}\boldsymbol{\mathrm{M}}_{t}(X),\boldsymbol{\mathrm{PL}}(\boldsymbol{\mathrm{M}}_{s}(X))\Big{)}$
(6) s.t.
$\displaystyle\quad\boldsymbol{\mathrm{M}}_{t}(\boldsymbol{\mathrm{T}}(X))\approx\boldsymbol{\mathrm{M}}_{t}(X)\;\;\;\;\;\;\;\;\forall
X\in\mathcal{D}_{t},\boldsymbol{\mathrm{T}}\in\mathcal{T}_{a}$ (7)
$\displaystyle\quad\boldsymbol{\mathrm{M}}_{t}(\boldsymbol{\mathrm{T}}(X))\approx\boldsymbol{\mathrm{T}}(\boldsymbol{\mathrm{M}}_{t}(X))\;\;\;\forall
X\in\mathcal{D}_{t},\boldsymbol{\mathrm{T}}\in\mathcal{T}_{s}$ (8)

The objective function is thus to optimize for the pseudo-labels for the
confident pixels as obtained from Eqn. 3 using cross-entropy loss $l_{c}$,
such that the output of the target model is consistent with augment and
spatial transforms. In practice, we compose the two sets of transforms to
strengthen its power. To do so, in each iteration, we randomly choose a set of
transformations
$\mathcal{T}=\\{\boldsymbol{\mathrm{T}}_{0},\dots,\boldsymbol{\mathrm{T}}_{k}\\}\in\mathcal{P}(\mathcal{T}_{a}\cup\mathcal{T}_{s})\setminus\emptyset$.
Due to the differences in the consistencies, the transforms need to be
composed differently for input and output. The input transform is a
composition of all the transforms,
$\boldsymbol{\mathrm{T}}_{i}=\boldsymbol{\mathrm{T}}_{0}\circ\dots\circ\boldsymbol{\mathrm{T}}_{k}$.
But to compose the output transform, we remove those drawn from
$\mathcal{T}_{a}$, and then compose the rest in the same manner as input
transform to obtain $\boldsymbol{\mathrm{T}}_{o}$. Then, the constraints can
be simplified as
$\boldsymbol{\mathrm{M}}_{t}(\boldsymbol{\mathrm{T}}_{i}(X))\approx\boldsymbol{\mathrm{T}}_{o}(\boldsymbol{\mathrm{M}}_{t}(X))$.

We approximate the constraint using two losses, and impose it via the penalty
method. For every iteration, we consider
$\boldsymbol{\mathrm{T}}_{o}(\boldsymbol{\mathrm{M}}_{t}(X))$ i.e., the
transformed outputs from the current iteration target model, as ground-truths
for the losses, and do not pass gradients through it, as otherwise trivial
solutions can satisfy the constraints. The two losses are:

$\displaystyle\mathcal{L}_{r}$
$\displaystyle=\sum_{X\in\mathcal{D}_{t}}l_{c}\Big{(}\boldsymbol{\mathrm{M}}_{t}(\boldsymbol{\mathrm{T}}_{i}(X)),\boldsymbol{\mathrm{T}}_{o}(\boldsymbol{\mathrm{M}}_{t}(X))\Big{)}$
$\displaystyle\quad\quad\quad\quad+l_{c}\Big{(}\boldsymbol{\mathrm{M}}_{t}(\boldsymbol{\mathrm{T}}_{i}(X)),\boldsymbol{\mathrm{T}}_{o}(\boldsymbol{\mathrm{PL}}(\boldsymbol{\mathrm{M}}_{t}(X)))\Big{)}$
(9)

where $l_{c}$ is the cross-entropy loss. The first loss is a form of soft
pseudo-labeling and computed for every pixel, whereas the second loss is hard
pseudo-labeling, which only computes the loss for the confident pixels.

Parameter Updates. We optimize using Stochastic Gradient Descent. Note that we
use the pseudo-labels using $\boldsymbol{\mathrm{M}}_{s}$ for ground-truth in
the loss function of Eqn. 6 and $\boldsymbol{\mathrm{M}}_{t}$ in the current
iteration as the ground-truth for the loss function in Eqn. 9. For clarity,
the SGD updates for the $k^{th}$ iteration, with $\eta$ as the learning rate
can be expressed as follows:

$\boldsymbol{\mathrm{M}}_{t}^{k+1}=\boldsymbol{\mathrm{M}}_{t}^{k}-\eta\nabla_{\boldsymbol{\mathrm{M}}_{t}}\Big{(}\mathcal{L}_{c}(\boldsymbol{\mathrm{M}}_{s})+\mathcal{L}_{r}(\boldsymbol{\mathrm{M}}_{t}^{k})\Big{)}$
(10)

where the argument in the brackets represent the models used to obtain the
ground truths to compute the losses.

Pseudo-label Thresholds.

To obtain the pseudo-labels using the source model $\mathrm{M}_{s}$, we set
the thresholds to be $\min(0.9,\text{median of label-wise confidence})$, where
the median is computed over all pixel predictions of the dataset for that
label. The confidence distribution varies across labels, which motivates us to
choose a label-dependent threshold rather than a fixed threshold for all
labels. This strategy is commonly used in literature, and we also observe a
boost in performance using this strategy. Now, as the target model
$\boldsymbol{\mathrm{M}}_{t}$ evolves over time, we do not keep the same
threshold to obtain the pseudo-label for its predictions, but rather update it
using exponential moving average as follows:

$\boldsymbol{\mathrm{p}}=\lambda\boldsymbol{\mathrm{p}}+(1-\lambda)\boldsymbol{\mathrm{p}}_{k}$
(11)

where $\boldsymbol{\mathrm{p}}$ is the threshold vector over the $C$
categories and used for the target model, $\boldsymbol{\mathrm{p}}_{k}$ is the
threshold vector obtained for the image in the $k^{th}$ iteration using the
median strategy mentioned above, computed over all pixel predictions of that
image for that label.

##  4 Experiments

Table 1: Results of adapting GTA5 to Cityscapes. The top group are methods
which use source data during adaptation, while the bottom two row groups do
not use any source data to adapt. Note that the middle row group use
DeepLabv3+ResNet-50 as their backbone while all other methods use
DeepLabv2+ResNet-101.

Source | Method | Stuff | Things | mIoU
---|---|---|---|---
Yes |  AdaptOutput [50] | 52.2 | 33.6 | 41.4
AdvEnt [54] | 57.0 | 37.1 | 45.5
SSF-DAN [12] | 57.4 | 36.7 | 45.4
BDL [25] | 61.7 | 38.9 | 48.5
CAG [63] | 61.2 | 42.0 | 50.2
WeakDA [39] | 61.0 | 38.8 | 48.2
Stuff [56] | 62.1 | 39.9 | 49.2
|  FDA [59] | 60.3 | 43.2 | 50.4
|  SAC [1] | 64.3 | 46.1 | 53.8
No |  SFDA [28] | 56.7 | 33.3 | 43.2
No | No Adapt. | 45.6 | 31.6 | 37.5
URMA [42] | 60.3 | 34.0 | 45.1
LD[61] | 60.1 | 34.9 | 45.5
| Ours | 59.0 | 41.3 | 48.8

Table 2:  Results of adapting SYNTHIA to Cityscapes. The top group are methods
which use source data to adapt, while the bottom two row groups do not use
source data to adapt. mIoU and mIoU∗ are averaged over 16 and 13 categories.
Note that the middle row group use DeepLabv3+ResNet-50 as the backbone while
all other methods use DeepLabv2+ResNet-101. Stuff and Things mIoU is computed
over the 13 categories for which all works report.

Source | Method | Stuff | Things | mIoU |  mIoU∗
---|---|---|---|---|---
Yes |  AdaptOutput [50] | 70.8 | 30.4 | 39.5 | 45.9
AdvEnt [54] | 74.4 | 31.5 | 41.2 | 48.0
SSF-DAN [12] | 74.6 | 34.6 | - | 50.0
CAG [63] | 73.9 | 37.4 | 44.5 | 51.4
WeakDA [39] | 78.5 | 35.3 | 44.3 | 51.9
Stuff [56] | 73.9 | 38.5 | - | 52.1
FDA [59] | 66.4 | 43.8 | - | 52.5
|  SAC [1] | 79.7 | 46.5 | 52.6 | 59.3
No |  SFDA [28] | 76.8 | 26.5 | 39.2 | 45.9
No | No Adapt. | 57.0 | 23.9 | 32.1 | 36.7
URMA [42] | 64.9 | 32.6 | 39.6 | 45.0
LD[61] | 70.0 | 37.7 | 42.6 | 50.1
| Ours | 69.7 | 37.9 | 43.7 | 50.2

Datasets:

We evaluate our framework on three combinations of source $\rightarrow$ target
datasets covering both outdoor and indoor scenes. For outdoor, we evaluate on
GTA5 [40] $\rightarrow$ Cityscapes [9] and SYNTHIA [41] $\rightarrow$
Cityscapes. For indoor, we use SceneNet [33] $\rightarrow$ SUN [46]. Note that
unlike existing literature, this work is the first to show results on the
indoor setting, which depicts the generalization ability of our framework. The
outdoor scene dataset has $19$ categories, whereas the indoor has $13$
categories. Please refer to Appendix A for more details.

Implementation Details:

To have a fair comparison with the works in literature, we use the Deeplab-V2
[5] with ResNet-101 [14] as the network. We use one GPU to train our models
with a batch size of $1$ in all experiments. We use SGD with an initial
learning rate of $2.5\times 10^{-4}$ with polynomial decay of power $0.9$ [5].
We use the standard metric of mean intersection over union (mIoU) [5] to
evaluate all algorithms.

Table 3:  Results of adapting SceneNet to SUN. The top row group does not use
any source data to adapt, while the bottom row uses full supervision on the
target images.

Method |  bed |  books |  ceiling |  chair |  floor |  furniture |  objects |  picture |  sofa |  table |  tv |  wall |  window | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
No Adapt. | 19.6 | 10.1 | 22.2 | 42.5 | 65.4 | 21.3 | 13.4 | 20.9 | 18.2 | 27.1 | 6.2 | 57.1 | 20.2 | 26.5
Ours | 35.3 | 23.5 | 34.1 | 48.7 | 73.6 | 26.7 | 11.1 | 29.9 | 36.9 | 38.1 | 15.0 | 63.2 | 27.2 | 35.6
Full Supervision | 53.1 | 32.6 | 54.0 | 60.0 | 82.4 | 35.1 | 33.4 | 43.2 | 45.7 | 52.7 | 36.8 | 72.0 | 46.8 | 49.8

Table 4: Ablation of the transformations on GTA5 $\rightarrow$ Cityscapes.

Collage | Mirror | Rotate | Gaussian | Cutout | mIoU
---|---|---|---|---|---
✓ |  |  |  |  | 46.4
| ✓ |  |  |  | 45.9
|  | ✓ |  |  | 45.5
|  |  | ✓ |  | 46.4
|  |  |  | ✓ | 45.8
✓ | ✓ | ✓ | ✓ | ✓ | 48.8

Comparison with state-of-the-art:

We first compare our method with state-of-the-art in literature in Table 7 for
GTA5 $\rightarrow$ Cityscapes, in Table 8 for SYNTHIA $\rightarrow$
Cityscapes, and SceneNet $\rightarrow$ SUN in Table 4. Due to limited space,
instead of presenting the category-wise performances, we club them into COCO-
style Stuff and Things in Table 7 and 8, and present the category-wise results
in appendix (Table 7 and Table 8). Stuff includes road, sidewalk, building,
wall, fence, veg, terrain and sky; while Things include sign, person, rider,
car, truck, bus, train, mbike, bike, light and pole. “No Adapt.” means
applying the source model directly on the target data, without adaptation.
Also, as this is the first work which presents domain adaptation results on
SceneNet $\rightarrow$ SUN, we include the result with full-supervision on the
target, in Table 4. As we can see, our method performs much better than “No
Adapt.”, and is only a few points short of the state-of-the-art methods which
use source data for adaptation.

From the tables we can also see that, our method outperforms other source-free
adaptation methods, even with just using a batch size of $1$, whereas some
baselines use batch size $>1$ to train. Our method is also comparable to many
recent methods, which use source data to adapt. Most methods in literature
perform an alignment either in the input or output space between the source
and target domain, which not only requires the source data, but also
additional models (discriminators), which are trained simultaneously. Compared
to that, our framework is quite light-weight as it just involves pseudo-label
training along with certain constraints.

Ablation study of transforms:

In this experiment, we analyse the effect of transforms we use in our
framework and the effect of learning from collage images. We use two spatial
transforms, Mirror and Rotate and two augment transforms, Gaussian filter and
Cutout. The results are presented in Table 4. We evaluate the model by adding
only one transform at a time. As can be seen, the transforms individually
improve the performance beyond just pseudo-labeling. Moreover, learning from
collage images also improves the performance beyond pseudo-label learning from
single images. The maximum gain is achieved when we use all of the above
together. We also see similar trends for other datasets as well, and present
them in appendix (Table 9). It is interesting to note that augment transforms
beyond a certain parameter limit do not work as well, which is intuitive, as
the content of the image may change beyond what is necessary for fine
segmentation. For example, in Gaussian filtering, using a filter size more
than $20$ does not help with an image of size $512\times 1024$.

Ablation study of the framework:

In this experiment, we break down every part of the framework and evaluate
their performance. We present the results in Table 5 for all the datasets.
When we first update the normalization parameters, the performance improves by
$1-4\%$. Then using the updated network for pseudo-label training using the
target images offers a further $4-5\%$ improvement. Now, imposing the
transform constraints using the hard and soft constraint losses as in Eqn. 9
along with the collage images further improves the performance by about $3\%$
compared to just pseudo-labeling.

Ablation of design choices:

We perform ablation of various choices involved in designing our proposed
algorithm. The first one is using category-wise thresholds for pseudo-
labeling, compared to uniform thresholding for all labels. To compare, we
execute our framework with an uniform threshold of $0.9$ for all the labels
(following [25]), as mentioned in the sixth row in Table 5. We observe that
using label-wise thresholding (“Ours” in Table 5) performs better by $1.5-4\%$
than uniform thresholding.

Table 5:  Ablation of loss functions and design choices.

Methods | GTA5 | SYNTHIA | SceneNet
---|---|---|---
$\rightarrow$ Cityscapes |  $\rightarrow$ Cityscapes |  $\rightarrow$ SUN
No Adapt. | 37.5 | 32.1 | 26.5
Update Norm | 41.4 | 35.3 | 27.4
Pseudo-Label (PL) | 45.6 | 40.2 | 32.1
PL + soft constraints | 48.2 | 43.1 | 35.0
PL + hard constraints | 47.9 | 43.0 | 34.8
Uniform thresholds | 47.3 | 41.6 | 31.9
Constraints using $\boldsymbol{\textrm{M}}_{s}$ | 47.3 | 42.2 | 33.2
Finetuning $\boldsymbol{\textrm{M}}_{s}$ | 48.6 | 43.3 | 28.9
PL + augment | 43.1 | 39.1 | 33.8
Ours | 48.8 | 43.7 | 35.6

In the next ablation, we study the effect of using the norm updated source
model $\boldsymbol{\textrm{M}}_{s}$ instead of target model
$\boldsymbol{\textrm{M}}_{t}$ in current iteration, for the constraints of
Eqn. 7, 8. If we apply the transform constraints using
$\boldsymbol{\textrm{M}}_{s}$ rather than $\boldsymbol{\textrm{M}}_{t}$, then
we limit its ability to improve beyond the fixed pseudo-labels, albeit with
transforms applied on them. In other words, when using
$\boldsymbol{\mathrm{M}}_{t}$ for the constraints, the optimization process is
allowed to learn the target model and self-improve in a way such that the
constraints are satisfied on the final target model. This helps to improve the
performance, as is evident by comparing the seventh row in Table 5 with the
last row.

Next, we investigate whether to fine-tune the source model for adaptation or
train a new target model from scratch. As can be observed from the eighth row
of Table 5, fine-tuning performs worse than our method where we train a new
target model from scratch. Note that the learning rate and the number of
training epochs are kept the same for both cases. This can be attributed to
the source model being already in a local optima in the loss landscape, thus
may be less influenced by the pseudo-label losses and constraints.

Finally, we use the transforms in the standard augmentation setting (but
stronger than normal augments), i.e., augment the images, pseudo-label them
and learn the target model using them. We use equal portion of original and
augmented images, as in our method. The results are presented as “PL+augment”
in Table 5, which shows that our consistency based approach performs much
better. This is because the pseudo-labels are much better on the original
images rather than on the augmented ones, and it is better to transform the
pseudo-labeled image, rather than pseudo-labeling the transformed image.

Table 6: Results for test-time adaptation with a single iteration of optimization at test time.  Methods | GTA5 | SYNTHIA | SceneNet
---|---|---|---
$\rightarrow$ Cityscapes |  $\rightarrow$ Cityscapes |  $\rightarrow$ SUN
No Adapt. | 37.5 | 32.1 | 26.5
Entropy [55] | 38.4 | 32.6 | 27.3
Likelihood [35] | 38.3 | 32.5 | 27.2
Ours | 39.3 | 33.2 | 27.6

Test Time Adaptation:

To further validate the effectiveness of our consistency based self-training
approach, we use it for test-time adaptation, which is recently introduce in
[48]. Here, we do not even have any unlabeled target data to adapt. Instead,
we use our loss to individually adapt to each test image, resetting the model
back to the source model after predicting. For TTA, we do not use the collage
transformations, as we adapt to a single image at a time. We show results on
the episodic fully test time adaptation setting proposed by [55] where the
network cannot be trained again with any additional source or target data,
other than the given test image. We compare our loss function with those from
recent works [35, 55] and observe that our method performs better due to its
inherent ability to exploit the regularization provided by semantic
segmentation as a task (Table 10). All results share the same backbone and
hyper-parameters as described in implementation details. For [35], we report
the best result from SLR/HLR loss variants.

Figure 5: Examples from indoor and outdoor scenarios for visual comparison.
The source$\rightarrow$target datasets are mentioned at the top of the
columns. “No Adapt” refers to using the source model directly for prediction,
without any adaptation.

Qualitative Analysis:

We present a few images and their segmentation maps for visual analysis in
Figure 5. We compare our method with “No Adapt”, i.e., directly applying the
source model. As we can see in the left image, our method is able to discover
one of the signs and the sidewalk, which is not segmented in “No Adapt”. For
the indoor case, our method correctly segmented the glass regions as window
(light red), which is segmented as picture (yellow) in “No Adapt”. In the last
column, the signs are correctly discovered by our algorithm, and the road is
properly segmented, while it is labeled as car (blue) in “No Adapt”. Overall,
the predictions made by our method are smooth with much less noise, which can
potentially be attributed to the consistency constraints. We present more
examples in appendix (Figure 8).

##  5 Conclusion

We propose a framework for adapting semantic segmentation models from source
to target without access to the labeled source data. We present a self-
training framework by enforcing consistencies with certain transforms to
efficiently extract information from the source model. With only a few
existing works in this challenging setting, our approach compares favorably
against strong baselines. Empirical evaluation shows that our method performs
comparable to many state-of-the-art methods that use source data to adapt. We
further show the usefulness of our approach for fully test-time adaptation in
which adaptation is done individually for each test image. Future works can
explore extracting information such as shape priors, from the source and
infusing them into the target model, for better adaptation.

## References

  * [1] Nikita Araslanov and Stefan Roth.  Self-supervised augmentation consistency for adapting semantic segmentation.  In CVPR, 2021.
  * [2] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan.  Unsupervised pixel-level domain adaptation with generative adversarial networks.  In CVPR, 2017.
  * [3] Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han.  Domain-specific batch normalization for unsupervised domain adaptation.  In CVPR, pages 7354–7362, 2019.
  * [4] Wei-Lun Chang, Hui-Po Wang, Wen-Hsiao Peng, and Wei-Chen Chiu.  All about structure: Adapting structural information across domains for boosting semantic segmentation.  In CVPR, 2019.
  * [5] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  TPAMI, 2017.
  * [6] Yuhua Chen, Wen Li, and Luc Van Gool.  Road: Reality oriented adaptation for semantic segmentation of urban scenes.  In CVPR, 2018.
  * [7] Yi-Hsin Chen, Wei-Yu Chen, Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang, and Min Sun.  No more discrimination: Cross city adaptation of road scene segmenters.  In ICCV, 2017.
  * [8] Jaehoon Choi, Taekyung Kim, and Changick Kim.  Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation.  In ICCV, 2019.
  * [9] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [10] Shuyang Dai, Kihyuk Sohn, Yi-Hsuan Tsai, Lawrence Carin, and Manmohan Chandraker.  Adaptation across extreme variations using unlabeled domain bridges.  BMVC, 2020.
  * [11] Jiahua Dong, Yang Cong, Gan Sun, Yuyang Liu, and Xiaowei Xu.  Cscl: Critical semantic-consistent learning for unsupervised domain adaptation.  In ECCV, 2020.
  * [12] Liang Du, Jingang Tan, Hongye Yang, Jianfeng Feng, Xiangyang Xue, Qibao Zheng, Xiaoqing Ye, and Xiaolin Zhang.  Ssf-dan: Separated semantic feature based domain adaptation network for semantic segmentation.  In ICCV, 2019.
  * [13] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  In JMLR, 2016.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [15] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In ICML, 2018.
  * [16] Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell.  Fcns in the wild: Pixel-level adversarial and constraint-based adaptation.  CoRR, abs/1612.02649, 2016.
  * [17] Jiaxing Huang, Shijian Lu, Dayan Guan, and Xiaobing Zhang.  Contextual-relation consistent domain adaptation for semantic segmentation.  In ECCV, 2020.
  * [18] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In ICML, 2015.
  * [19] Myeongjin Kim and Hyeran Byun.  Learning texture invariant representation for domain adaptation of semantic segmentation.  In CVPR, 2020.
  * [20] Jogendra Nath Kundu, Akshay Kulkarni, Amit Singh, Varun Jampani, and R Venkatesh Babu.  Generalize then adapt: Source-free domain adaptive semantic segmentation.  In ICCV, 2021.
  * [21] Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht.  Sliced wasserstein discrepancy for unsupervised domain adaptation.  In CVPR, 2019.
  * [22] Guangrui Li, Guoliang Kang, Wu Liu, Yunchao Wei, and Yi Yang.  Content-consistent matching for domain adaptive semantic segmentation.  In ECCV, 2020.
  * [23] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu.  Model adaptation: Unsupervised domain adaptation without source data.  In CVPR, pages 9641–9650, 2020.
  * [24] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou.  Revisiting batch normalization for practical domain adaptation.  ICLRW, 2017.
  * [25] Yunsheng Li, Lu Yuan, and Nuno Vasconcelos.  Bidirectional learning for domain adaptation of semantic segmentation.  In CVPR, 2019.
  * [26] Qing Lian, Fengmao Lv, Lixin Duan, and Boqing Gong.  Constructing self-motivated pyramid curriculums for cross-domain semantic segmentation: A non-adversarial approach.  In ICCV, 2019.
  * [27] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In ICML, 2020.
  * [28] Yuang Liu, Wei Zhang, and Jun Wang.  Source-free domain adaptation for semantic segmentation.  In CVPR, 2021.
  * [29] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In ICML, 2015.
  * [30] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  In NIPS, 2016.
  * [31] Fengmao Lv, Tao Liang, Xiang Chen, and Guosheng Lin.  Cross-domain semantic segmentation via domain-invariant interactive relation transfer.  In CVPR, 2020.
  * [32] John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J Davison.  Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth.  arXiv preprint arXiv:1612.05079, 2016.
  * [33] John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J Davison.  Scenenet rgb-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation?  In ICCV, 2017.
  * [34] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang.  Instance adaptive self-training for unsupervised domain adaptation.  ECCV, 2020.
  * [35] Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen.  Test-time adaptation to distribution shift by confidence maximization and input transformation.  arXiv preprint arXiv:2106.14999, 2021.
  * [36] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim.  Image to image translation for domain adaptation.  In CVPR, 2018.
  * [37] Luigi Musto and Andrea Zinelli.  Semantically adaptive image-to-image translation for domain adaptation of semantic segmentation.  BMVC, 2020.
  * [38] Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon.  Unsupervised intra-domain adaptation for semantic segmentation through self-supervision.  In CVPR, 2020.
  * [39] Sujoy Paul, Yi-Hsuan Tsai, Samuel Schulter, Amit K Roy-Chowdhury, and Manmohan Chandraker.  Domain adaptive semantic segmentation using weak labels.  ECCV, 2020.
  * [40] Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In ECCV, 2016.
  * [41] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez.  The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In CVPR, 2016.
  * [42] Prabhu Teja S and Francois Fleuret.  Uncertainty reduction for model adaptation in semantic segmentation.  In CVPR, 2021.
  * [43] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In CVPR, 2018.
  * [44] Fatemeh Sadat Saleh, Mohammad Sadegh Aliakbarian, Mathieu Salzmann, Lars Petersson, and Jose M. Alvarez.  Effective use of synthetic data for urban scene semantic segmentation.  In ECCV, 2018.
  * [45] Inkyu Shin, Sanghyun Woo, Fei Pan, and In So Kweon.  Two-phase pseudo label densification for self-training based domain adaptation.  In ECCV, 2020.
  * [46] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In CVPR, 2015.
  * [47] M Naseer Subhani and Mohsen Ali.  Learning from scale-invariant examples for domain adaptation in semantic segmentation.  ECCV, 2020.
  * [48] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In ICML, 2020.
  * [49] Luan Tran, Kihyuk Sohn, Xiang Yu, Xiaoming Liu, and Manmohan Chandraker.  Gotta adapt ’em all: Joint pixel and feature-level domain adaptation for recognition in the wild.  In CVPR, 2019.
  * [50] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker.  Learning to adapt structured output space for semantic segmentation.  In CVPR, 2018.
  * [51] Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, and Manmohan Chandraker.  Domain adaptation for structured output via discriminative patch representations.  In ICCV, 2019.
  * [52] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In CVPR, 2017.
  * [53] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  arXiv preprint arXiv:1607.08022, 2016.
  * [54] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation.  In CVPR, 2019.
  * [55] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In ICLR, 2021.
  * [56] Zhonghao Wang, Mo Yu, Yunchao Wei, Rogerio Feris, Jinjun Xiong, Wen-mei Hwu, Thomas S Huang, and Honghui Shi.  Differential treatment for stuff and things: A simple unsupervised domain adaptation method for semantic segmentation.  In CVPR, 2020.
  * [57] Zuxuan Wu, Xintong Han, Yen-Liang Lin, Mustafa Gkhan Uzunbas, Tom Goldstein, Ser Nam Lim, and Larry S. Davis.  Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation.  In ECCV, 2018.
  * [58] Haifeng Xia, Handong Zhao, and Zhengming Ding.  Adaptive adversarial network for source-free domain adaptation.  In ICCV, 2021.
  * [59] Yanchao Yang and Stefano Soatto.  Fda: Fourier domain adaptation for semantic segmentation.  In CVPR, 2020.
  * [60] Hao-Wei Yeh, Baoyao Yang, Pong C Yuen, and Tatsuya Harada.  Sofa: Source-data-free feature alignment for unsupervised domain adaptation.  In WACV, 2021.
  * [61] Fuming You, Jingjing Li, Lei Zhu, Zhi Chen, and Zi Huang.  Domain adaptive semantic segmentation without source data.  In ACM-MM, 2021.
  * [62] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  ICLR, 2017.
  * [63] Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao.  Category anchor-guided unsupervised domain adaptation for semantic segmentation.  NeurIPS, 2019.
  * [64] Yang Zhang, Philip David, and Boqing Gong.  Curriculum domain adaptation for semantic segmentation of urban scenes.  In ICCV, 2017.
  * [65] Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei.  Fully convolutional adaptation networks for semantic segmentation.  In CVPR, 2018.
  * [66] Yang Zou, Zhiding Yu, B. V. K. Vijaya Kumar, and Jinsong Wang.  Domain adaptation for semantic segmentation via class-balanced self-training.  In ECCV, 2018.

##  Appendix A Datasets

GTA5 $\rightarrow$ Cityscapes:

In this setting, we consider GTA5 [40] as the source and Cityscapes [9] as the
target dataset. The source images are at $760\times 1280$ resolution, while
the target images are used at $512\times 1024$. The datasets have $19$
categories. The source dataset has $24966$ training images and the target has
$2975$ training images on which we perform adaptation, and $500$ validation
images. We report the performance on these $500$ validation images as is the
usual practice in the literature.

SYNTHIA $\rightarrow$ Cityscapes:

In this setting, we consider SYNTHIA [41] as the souce and Cityscapes [9] as
the target dataset. SYNTHIA contains $9400$ training images. However, unlike
GTA5, due to lack of proper annotations for a few categories in SYNTHIA, we
remove them from evaluation and report the results for $16$ categories,
following the literature.

SceneNet $\rightarrow$ SUN:

Both of the above two settings are for outdoor scenes, and in this setting we
consider indoor scenes with SceneNet [32] as the source and SUN [46] as the
target. The SceneNet dataset has around $5$ million simulated images. However,
a lot of the images are rather simple with only a few categories in them.
Thus, to train the source model, we only choose the top $50,000$ images having
the highest number of categories. The SUN dataset contains $5285$ training
images on which we perform adaptation and $5050$ test images which are used
for evaluation. Both of these datasets contain $13$ categories. Specifically,
we use the label transformation available with the SceneNet dataset
111https://github.com/ankurhanda/sunrgbd-meta-data to map the labels in the
SUN dataset such that it matches with the label space of SceneNet.

##  Appendix B Category-wise Segmentation Results

Table 7: Results of adapting GTA5 to Cityscapes. The top row group are methods
which use source data during adaptation, while the bottom row group do not use
any source data to adapt. Note that the middle row group use
DeepLabv3+ResNet-50 as their backbone while all other methods use
DeepLabv2+ResNet-101.

Source | Method |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  mbike |  bike | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Yes |  AdaptOutput [50] | 86.5 | 25.9 | 79.8 | 22.1 | 20.0 | 23.6 | 33.1 | 21.8 | 81.8 | 25.9 | 75.9 | 57.3 | 26.2 | 76.3 | 29.8 | 32.1 | 7.2 | 29.5 | 32.5 | 41.4
AdvEnt [54] | 89.4 | 33.1 | 81.0 | 26.6 | 26.8 | 27.2 | 33.5 | 24.7 | 83.9 | 36.7 | 78.8 | 58.7 | 30.5 | 84.8 | 38.5 | 44.5 | 1.7 | 31.6 | 32.4 | 45.5
SSF-DAN [12] | 90.3 | 38.9 | 81.7 | 24.8 | 22.9 | 30.5 | 37.0 | 21.2 | 84.8 | 38.8 | 76.9 | 58.8 | 30.7 | 85.7 | 30.6 | 38.1 | 5.9 | 28.3 | 36.9 | 45.4
BDL [25] | 91.0 | 44.7 | 84.2 | 34.6 | 27.6 | 30.2 | 36.0 | 36.0 | 85.0 | 43.6 | 83.0 | 58.6 | 31.6 | 83.3 | 35.3 | 49.7 | 3.3 | 28.8 | 35.6 | 48.5
CAG [63] | 90.4 | 51.6 | 83.8 | 34.2 | 27.8 | 38.4 | 25.3 | 48.4 | 85.4 | 38.2 | 78.1 | 58.6 | 34.6 | 84.7 | 21.9 | 42.7 | 41.1 | 29.3 | 37.2 | 50.2
WeakDA [39] | 91.6 | 47.4 | 84.0 | 30.4 | 28.3 | 31.4 | 37.4 | 35.4 | 83.9 | 38.3 | 83.9 | 61.2 | 28.2 | 83.7 | 28.8 | 41.3 | 8.8 | 24.7 | 46.4 | 48.2
Stuff [56] | 90.6 | 44.7 | 84.8 | 34.3 | 28.7 | 31.6 | 35.0 | 37.6 | 84.7 | 43.3 | 85.3 | 57.0 | 31.5 | 83.8 | 42.6 | 48.5 | 1.9 | 30.4 | 39.0 | 49.2
|  FDA [59] | 92.5 | 53.3 | 82.3 | 26.5 | 27.6 | 36.4 | 40.5 | 38.8 | 82.2 | 39.8 | 78.0 | 62.6 | 34.4 | 84.9 | 34.1 | 53.1 | 16.8 | 27.7 | 46.4 | 50.4
|  SAC [1] | 90.4 | 53.9 | 86.6 | 42.4 | 27.3 | 45.1 | 48.5 | 42.7 | 87.4 | 40.1 | 86.1 | 67.5 | 29.7 | 88.5 | 49.1 | 54.6 | 9.8 | 26.6 | 45.3 | 53.8
No |  SFDA [28] | 84.2 | 39.2 | 82.7 | 27.5 | 22.1 | 25.9 | 31.1 | 21.9 | 82.4 | 30.5 | 85.3 | 58.7 | 22.1 | 80.0 | 33.1 | 31.5 | 3.6 | 27.8 | 30.6 | 43.2
No | No Adapt. | 79.7 | 21.8 | 66.8 | 19.3 | 20.6 | 22.8 | 28.9 | 12.9 | 76.3 | 19.5 | 60.9 | 56.2 | 26.6 | 77.8 | 33.3 | 27.7 | 3.9 | 25.0 | 32.5 | 37.5
URMA [42] | 92.3 | 55.2 | 81.6 | 30.8 | 18.8 | 37.1 | 17.7 | 12.1 | 84.2 | 35.9 | 83.8 | 57.7 | 24.1 | 81.7 | 27.5 | 44.3 | 6.9 | 24.1 | 40.4 | 45.1
LD[61] | 91.6 | 53.2 | 80.6 | 36.6 | 14.2 | 26.4 | 31.6 | 22.7 | 83.1 | 42.1 | 79.3 | 57.3 | 26.6 | 82.1 | 41.0 | 50.1 | 0.3 | 25.9 | 19.5 | 45.5
| Ours | 89.2 | 37.3 | 82.4 | 29.0 | 23.5 | 31.8 | 34.6 | 28.7 | 84.8 | 45.5 | 80.2 | 62.6 | 32.6 | 86.1 | 45.6 | 43.8 | 0.0 | 34.6 | 54.4 | 48.8

Table 8:  Results of adapting SYNTHIA to Cityscapes. The top group are methods
which use source data during adaptation, while the bottom row do not use any
source data to adapt. mIoU and mIoU∗ are averaged over 16 and 13 categories.
Note that the middle row group use DeepLabv3+ResNet-50 as their backbone while
all other methods use DeepLabv2+ResNet-101.

Source | Method |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  sky |  person |  rider |  car |  bus |  mbike |  bike | mIoU |  mIoU∗
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Yes |  AdaptOutput [50] | 79.2 | 37.2 | 78.8 | 10.5 | 0.3 | 25.1 | 9.9 | 10.5 | 78.2 | 80.5 | 53.5 | 19.6 | 67.0 | 29.5 | 21.6 | 31.3 | 39.5 | 45.9
AdvEnt [54] | 85.6 | 42.2 | 79.7 | 8.7 | 0.4 | 25.9 | 5.4 | 8.1 | 80.4 | 84.1 | 57.9 | 23.8 | 73.3 | 36.4 | 14.2 | 33.0 | 41.2 | 48.0
SSF-DAN [12] | 84.6 | 41.7 | 80.8 | - | - | - | 11.5 | 14.7 | 80.8 | 85.3 | 57.5 | 21.6 | 82.0 | 36.0 | 19.3 | 34.5 | - | 50.0
CAG [63] | 84.7 | 40.8 | 81.7 | 7.8 | 0.0 | 35.1 | 13.3 | 22.7 | 84.5 | 77.6 | 64.2 | 27.8 | 80.9 | 19.7 | 22.7 | 48.3 | 44.5 | 51.4
WeakDA [39] | 92.0 | 53.5 | 80.9 | 11.4 | 0.4 | 21.8 | 3.8 | 6.0 | 81.6 | 84.4 | 60.8 | 24.4 | 80.5 | 39.0 | 26.0 | 41.7 | 44.3 | 51.9
|  Stuff [56] | 83.0 | 44.0 | 80.3 | - | - | - | 17.1 | 15.8 | 80.5 | 81.8 | 59.9 | 33.1 | 70.2 | 37.3 | 28.5 | 45.8 | - | 52.1
|  FDA [59] | 79.3 | 35.0 | 73.2 | - | - | - | 19.9 | 24.0 | 61.7 | 82.6 | 61.4 | 31.1 | 83.9 | 40.8 | 38.4 | 51.1 | - | 52.5
|  SAC[1] | 89.3 | 47.2 | 85.5 | 26.5 | 1.3 | 43.0 | 45.5 | 32.0 | 87.1 | 89.3 | 63.6 | 25.4 | 86.9 | 35.6 | 30.4 | 53.0 | 52.6 | 59.3
No |  SFDA [28] | 81.9 | 44.9 | 81.7 | 4.0 | 0.5 | 26.2 | 3.3 | 10.7 | 86.3 | 89.4 | 37.9 | 13.4 | 80.6 | 25.6 | 9.6 | 31.3 | 39.20 | 45.9
No | No Adapt. | 37.6 | 18.7 | 73.8 | 9.95 | 0.1 | 26.4 | 8.9 | 13.9 | 74.7 | 80.4 | 52.4 | 16.1 | 39.2 | 21.9 | 13.2 | 25.8 | 32.1 | 36.7
URMA [42] | 59.3 | 24.6 | 77.0 | 14.0 | 1.8 | 31.5 | 18.3 | 32.0 | 83.1 | 80.4 | 46.3 | 17.8 | 76.7 | 17.0 | 18.5 | 34.6 | 39.6 | 45.0
LD[61] | 77.1 | 33.4 | 79.4 | 5.8 | 0.5 | 23.7 | 5.2 | 13.0 | 81.8 | 78.3 | 56.1 | 21.6 | 80.3 | 49.6 | 28.0 | 48.1 | 42.6 | 50.1
| Ours | 74.3 | 33.7 | 78.9 | 14.6 | 0.7 | 31.5 | 21.3 | 28.8 | 80.2 | 81.6 | 50.7 | 24.5 | 78.3 | 11.6 | 34.4 | 53.7 | 43.7 | 50.2

In this section, we present the category-wise segmentation results for the two
dataset settings, GTA5$\rightarrow$Cityscapes and
SYNTHIA$\rightarrow$Cityscapes in Table 7 and 8 respectively.

Figure 6: Ablation study of the cutout transformation.

##  Appendix C Hyper-parameter analysis of cutout

In this section, we analyse the effect of the hyper-parameters involved in the
cutout transformation. Recall from the paper, in this transformation, we
choose two parameters: size $b$ of the square block, and the percentage $p$ of
the image to be removed. Given these two parameters, we choose the number of
blocks to be removed as $k=\frac{pHW}{b^{2}}$. We execute our framework for
various values of $b$ and $p$, while keeping the rest of the framework same,
and present the performance obtained in Figure 6. By using this transform, we
want the network to learn rich context information, while performing
inpainting in the output space. As can be seen, with lower block size and
higher percentage of image removed, the performance degrades. This is because
by doing so the image becomes noisy, rather than structured removal, i.e.,
removed portions become well distributed throughout the image, which makes it
harder to learn context information. However, with higher block size, and
moderate percentage of image removed, the performance increases. We choose
$b=64,p=0.2$, in our experiments on outdoor images, where the resolution of
the image is high, and we choose $b=32,p=0.1$, for indoor experiments, with
lower image sizes.

Figure 7: Ablation study of the Gaussian filtering transformation. Figure 8:
Qualitative visualization. The first two columns and the last two columns show
results on outdoor scenes, while the middle two columns show results on indoor
scenes.

##  Appendix D Hyper-parameter analysis of Gaussian filter

In this section, we analyse the effect of two hyperparameters in the Gaussian
filter transformation. The two hyper-parameters are the kernel size and the
standard deviation of the filter. Instead of keeping a single standard
deviation for the filter, we choose it randomly between [0.1, $\sigma_{max}$],
where $\sigma_{max}$ is the hyper-parameter to choose. The image becomes more
blurry with higher kernel size and lower standard deviation. As the image
becomes more blurry, it becomes difficult for the network to figure out the
content, and thus we see a degradation in performance in the top right corner
in Figure 7. Note that the standard deviation mentioned in the figure signify
$\sigma_{max}$. We use a kernel size of $5$ and $\sigma_{max}=2$.

Table 9: Ablation of the transformations.  Collage | Mirror | Rotate | Gaussian | Cutout | SYNTHIA | SceneNet
---|---|---|---|---|---|---
|  |  |  |  |  $\rightarrow$ Cityscapes |  $\rightarrow$ SUN
✓ |  |  |  |  | 40.4 | 32.2
| ✓ |  |  |  | 41.9 | 34.6
|  | ✓ |  |  | 42.3 | 34.8
|  |  | ✓ |  | 42.5 | 34.5
|  |  |  | ✓ | 41.9 | 33.1
✓ | ✓ | ✓ | ✓ | ✓ | 43.7 | 35.6

##  Appendix E Ablation of the transformations

Here, we perform ablation of the four transformations used in our framework -
mirror, rotate, gaussian filter and cutout. In the paper, we presented results
on GTA5 $\rightarrow$ Cityscapes (Table 4), and here we present on the other
two settings, i.e., SYNTHIA $\rightarrow$ Cityscapes and SceneNet
$\rightarrow$ SUN. As can be seen in Table 9, each of the transformations have
their own contribution towards the final performance, however, an ensemble of
all the transformation performs the best.

##  Appendix F Additional test time adaptation results

In this section, we present additional results for the fully test time
adaptation setting, where we optimize the network for 5 iterations using the
given loss functions per single test image. Note that although not designed
specifically for test time adaptation, our consistency based approach is able
to perform good on this task as well. As can be seen in Table 10, we perform
better or within a close margin to the loss functions proposed by previous
works [55, 35] which were completely dedicated towards solving the test time
adaptation problem.

Table 10: Results for test-time adaptation with a single or five iterations of optimization at test time.  Methods | GTA5 | SYNTHIA | SceneNet
---|---|---|---
$\rightarrow$ Cityscapes |  $\rightarrow$ Cityscapes |  $\rightarrow$ SUN
No Adapt. | 37.6 | 32.1 | 26.5
1 Iteration
Entropy [55] | 38.4 | 32.6 | 27.3
Likelihood [35] | 38.3 | 32.5 | 27.2
Ours | 39.3 | 33.2 | 27.6
5 Iterations
Entropy [55] | 40.4 | 34.0 | 28.8
Likelihood [35] | 39.4 | 33.7 | 27.9
Ours | 42.2 | 36.0 | 28.4

##  Appendix G Qualitative Analysis

In Figure 8 we present a visual comparison of our method with directly
applying the source model on the target images, shown as “No Adapt”. As can be
seen in the first column, our method is able to properly label the signs, even
though the source model does not have them, as shown in “No Adapt”. Similar
discussions can be extended to results in the second column. In the third and
fourth columns, the source model is very confused with the shadows on the
chairs and tables, and assigns them multiple labels. However, our method is
able to predict the accurate labels with proper segmentation. In the last
column, the network mistakes most portion of the road as sidewalk, a prior
that pedestrians can be more often seen on sidewalks than roads. However, our
method is able segment the road and sidewalk properly and is also able to
segment a few of the road signs in the background.


* Ambainis and Emerson [2007] A. Ambainis and J. Emerson, Quantum t-designs: t-wise Independence in the Quantum World, in  _Twenty-Second Annual IEEE Conference on Computational Complexity (CCC’07)_ (2007) pp. 129–140.
  * Ho and Choi [2022] W. W. Ho and S. Choi, Exact Emergent Quantum State Designs from Quantum Chaotic Dynamics, Physical Review Letters 128, 060601 (2022).
  * Claeys and Lamacraft [2022] P. W. Claeys and A. Lamacraft, Emergent quantum state designs and biunitarity in dual-unitary circuit dynamics, Quantum 6, 738 (2022).
  * Gullans and Huse [2020a] M. J. Gullans and D. A. Huse, Dynamical Purification Phase Transition Induced by Quantum Measurements, Physical Review X 10, 041020 (2020a).
  * Skinner  _et al._ [2019] B. Skinner, J. Ruhman, and A. Nahum, Measurement-Induced Phase Transitions in the Dynamics of Entanglement, Physical Review X 9, 031009 (2019).
  * Li  _et al._ [2018] Y. Li, X. Chen, and M. P. A. Fisher, Quantum Zeno effect and the many-body entanglement transition, Physical Review B 98, 205136 (2018).
  * Li  _et al._ [2019] Y. Li, X. Chen, and M. P. A. Fisher, Measurement-driven entanglement transition in hybrid quantum circuits, Physical Review B 100, 134306 (2019).
  * Chan  _et al._ [2019] A. Chan, R. M. Nandkishore, M. Pretko, and G. Smith, Unitary-projective entanglement dynamics, Phys. Rev. B 99, 224307 (2019).
  * Hayden and Preskill [2007] P. Hayden and J. Preskill, Black holes as mirrors: quantum information in random subsystems, Journal of High Energy Physics 2007, 120 (2007).
  * Hosur  _et al._ [2016] P. Hosur, X.-L. Qi, D. A. Roberts, and B. Yoshida, Chaos in quantum channels, J. High Energ. Phys. 2, 4.
  * Roberts and Yoshida [2017] D. A. Roberts and B. Yoshida, Chaos and complexity by design, Journal of High Energy Physics 2017, 121 (2017).
  * Nahum  _et al._ [2018] A. Nahum, S. Vijay, and J. Haah, Operator Spreading in Random Unitary Circuits, Physical Review X 8, 021014 (2018).
  * von Keyserlingk  _et al._ [2018] C. W. von Keyserlingk, T. Rakovszky, F. Pollmann, and S. L. Sondhi, Operator Hydrodynamics, OTOCs, and Entanglement Growth in Systems without Conservation Laws, Physical Review X 8, 021013 (2018).
  * Mi  _et al._ [2021] X. Mi, P. Roushan, C. Quintana, S. Mandra, J. Marshall, C. Neill, F. Arute, K. Arya, _et al._ , Information scrambling in quantum circuits, Science 374, 1479 (2021).
  * Kim and Huse [2013] H. Kim and D. A. Huse, Ballistic Spreading of Entanglement in a Diffusive Nonintegrable System, Physical Review Letters 111, 127205 (2013).
  * Nahum  _et al._ [2017] A. Nahum, J. Ruhman, S. Vijay, and J. Haah, Quantum Entanglement Growth under Random Unitary Dynamics, Physical Review X 7, 031016 (2017).
  * Shenker and Stanford [2014] S. H. Shenker and D. Stanford, Black holes and the butterfly effect, Journal of High Energy Physics 2014, 67 (2014).
  * Knill  _et al._ [2008] E. Knill, D. Leibfried, R. Reichle, J. Britton, R. B. Blakestad, J. D. Jost, C. Langer, R. Ozeri, S. Seidelin, and D. J. Wineland, Randomized benchmarking of quantum gates, Phys. Rev. A 77, 012307 (2008).
  * Magesan  _et al._ [2011] E. Magesan, J. M. Gambetta, and J. Emerson, Scalable and robust randomized benchmarking of quantum processes, Phys. Rev. Lett. 106, 180504 (2011).
  * Brandão  _et al._ [2016] F. G. S. L. Brandão, A. W. Harrow, and M. Horodecki, Efficient quantum pseudorandomness, Phys. Rev. Lett. 116, 170502 (2016).
  * Alagic  _et al._ [2018] G. Alagic, T. Gagliardoni, and C. Majenz, Unforgeable quantum encryption, in  _Advances in Cryptology – EUROCRYPT 2018_ , edited by J. B. Nielsen and V. Rijmen (Springer International Publishing, Cham, 2018) pp. 489–519.
  * Bouland  _et al._ [2018] A. Bouland, B. Fefferman, C. Nirkhe, and U. Vazirani, On the complexity and verification of quantum random circuit sampling, Nature Physics 15, 159–163 (2018).
  * Haferkamp  _et al._ [2020] J. Haferkamp, D. Hangleiter, A. Bouland, B. Fefferman, J. Eisert, and J. Bermejo-Vega, Closing gaps of a quantum advantage with short-time hamiltonian dynamics, Phys. Rev. Lett. 125, 250501 (2020).
  * Sen [2006] P. Sen, Random measurement bases, quantum state distinction and applications to the hidden subgroup problem, in  _21st Annual IEEE Conference on Computational Complexity (CCC’06)_ (2006) pp. 14 pp.–287.
  * Hayden  _et al._ [2004] P. Hayden, D. Leung, P. W. Shor, and A. Winter, Randomizing quantum states: Constructions and applications, Communications in Mathematical Physics 250, 371–391 (2004).
  * Oszmaniec  _et al._ [2016] M. Oszmaniec, R. Augusiak, C. Gogolin, J. Kołodyński, A. Acín, and M. Lewenstein, Random bosonic states for robust quantum metrology, Phys. Rev. X 6, 041044 (2016).
  * Kimmel and Liu [2017] S. Kimmel and Y.-K. Liu, Phase retrieval using unitary 2-designs, 2017 International Conference on Sampling Theory and Applications (SampTA) 10.1109/sampta.2017.8024414 (2017).
  * Arute  _et al._ [2019] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. S. L. Brandao, D. A. Buell, and et al., Quantum supremacy using a programmable superconducting processor, Nature 574, 505–510 (2019).
  * Huang  _et al._ [2020] H.-Y. Huang, R. Kueng, and J. Preskill, Predicting many properties of a quantum system from very few measurements, Nature Physics 16, 1050 (2020).
  * Aharonov  _et al._ [2021] D. Aharonov, J. Cotler, and X.-L. Qi, Quantum Algorithmic Measurement, arXiv e-prints , arXiv:2101.04634 (2021), arXiv:2101.04634 [quant-ph] .
  * Huang  _et al._ [2021] H.-Y. Huang, R. Kueng, and J. Preskill, Information-theoretic bounds on quantum advantage in machine learning, Phys. Rev. Lett. 126, 190505 (2021).
  * Tran  _et al._ [2022] M. C. Tran, D. K. Mark, W. W. Ho, and S. Choi, Measuring Arbitrary Physical Properties in Analog Quantum Simulation 10.48550/arXiv.2212.02517 (2022).
  * McGinley and Fava [2022] M. McGinley and M. Fava, Shadow tomography from emergent state designs in analog quantum simulators 10.48550/arXiv.2212.02543 (2022).
  * Wilming and Roth [2022] H. Wilming and I. Roth, High-temperature thermalization implies the emergence of quantum state designs, arXiv:2202.01669 [cond-mat, physics:math-ph, physics:quant-ph] (2022).
  * Nandkishore and Huse [2015] R. Nandkishore and D. A. Huse, Many-Body Localization and Thermalization in Quantum Statistical Mechanics, Annual Review of Condensed Matter Physics 6, 15 (2015).
  * Abanin  _et al._ [2019] D. A. Abanin, E. Altman, I. Bloch, and M. Serbyn, Colloquium: Many-body localization, thermalization, and entanglement, Reviews of Modern Physics 91, 021001 (2019).
  * Altman  _et al._ [2021] E. Altman, K. R. Brown, G. Carleo, L. D. Carr, E. Demler, C. Chin, B. DeMarco, S. E. Economou, _et al._ , Quantum Simulators: Architectures and Opportunities, PRX Quantum 2, 017003 (2021).
  * Anza and Crutchfield [2021] F. Anza and J. P. Crutchfield, Beyond density matrices: Geometric quantum states, Physical Review A 103, 062218 (2021).
  * Anza and Crutchfield [2022] F. Anza and J. P. Crutchfield, Quantum Information Dimension and Geometric Entropy, PRX Quantum 3, 020355 (2022).
  * Goldstein  _et al._ [2006] S. Goldstein, J. L. Lebowitz, R. Tumulka, and N. Zanghi, On the Distribution of the Wave Function for Systems in Thermal Equilibrium, Journal of Statistical Physics 125, 1193 (2006).
  * Goldstein  _et al._ [2016] S. Goldstein, J. L. Lebowitz, C. Mastrodonato, R. Tumulka, and N. Zanghi, Universal Probability Distribution for the Wave Function of a Quantum System Entangled with its Environment, Communications in Mathematical Physics 342, 965 (2016).
  * [48] In a previous version of our manuscript, we provided the proof of monotonicity for the Schatten-indices $\alpha=1,2$ and $\infty$. We credit Daniel K. Mark for explaining to us how this result can be generalized to all $\alpha\geq 1$.
  * Low [2010] R. A. Low, Pseudo-randomness and Learning in Quantum Computation, arXiv e-prints 10.48550/arXiv.1006.5227 (2010).
  * Banuls  _et al._ [2009] M. C. Banuls, M. B. Hastings, F. Verstraete, and J. I. Cirac, Matrix Product States for Dynamical Simulation of Infinite Chains, Phys. Rev. Lett. 102, 240603 (2009).
  * Hastings and Mahajan [2015] M. B. Hastings and R. Mahajan, Connecting entanglement in time and space: Improving the folding algorithm, Phys. Rev. A 91, 032306 (2015).
  * Lerose  _et al._ [2021] A. Lerose, M. Sonner, and D. A. Abanin, Influence Matrix Approach to Many-Body Floquet Dynamics, Physical Review X 11, 021040 (2021).
  * Garratt and Chalker [2021a] S. J. Garratt and J. T. Chalker, Local Pairing of Feynman Histories in Many-Body Floquet Models, Physical Review X 11, 021051 (2021a).
  * Ippoliti and Khemani [2021] M. Ippoliti and V. Khemani, Postselection-Free Entanglement Dynamics via Spacetime Duality, Physical Review Letters 126, 060501 (2021).
  * Ippoliti  _et al._ [2022] M. Ippoliti, T. Rakovszky, and V. Khemani, Fractal, Logarithmic, and Volume-Law Entangled Nonthermal Steady States via Spacetime Duality, Physical Review X 12, 011045 (2022).
  * Lu and Grover [2021] T.-C. Lu and T. Grover, Spacetime duality between localization transitions and measurement-induced transitions, PRX Quantum 2, 040319 (2021).
  * Garratt and Chalker [2021b] S. J. Garratt and J. T. Chalker, Many-Body Delocalization as Symmetry Breaking, Physical Review Letters 127, 026802 (2021b).
  * Jonay  _et al._ [2021] C. Jonay, V. Khemani, and M. Ippoliti, Triunitary quantum circuits, Physical Review Research 3, 043046 (2021).
  * Bertini  _et al._ [2022] B. Bertini, K. Klobas, V. Alba, G. Lagnese, and P. Calabrese, Growth of R\’enyi Entropies in Interacting Integrable Models and the Breakdown of the Quasiparticle Picture, Physical Review X 12, 031016 (2022).
  * Akila  _et al._ [2016] M. Akila, D. Waltner, B. Gutkin, and T. Guhr, Particle-time duality in the kicked Ising spin chain, Journal of Physics A: Mathematical and Theoretical 49, 375101 (2016).
  * Bertini  _et al._ [2018] B. Bertini, P. Kos, and T. Prosen, Exact Spectral Form Factor in a Minimal Model of Many-Body Quantum Chaos, Physical Review Letters 121, 264101 (2018).
  * Gopalakrishnan and Lamacraft [2019] S. Gopalakrishnan and A. Lamacraft, Unitary circuits of finite depth and infinite width from quantum channels, Physical Review B 100, 064309 (2019).
  * Claeys and Lamacraft [2020] P. W. Claeys and A. Lamacraft, Maximum velocity quantum circuits, Physical Review Research 2, 033032 (2020).
  * Rather  _et al._ [2020] S. A. Rather, S. Aravinda, and A. Lakshminarayan, Creating Ensembles of Dual Unitary and Maximally Entangling Quantum Evolutions, Physical Review Letters 125, 070501 (2020).
  * Gutkin  _et al._ [2020] B. Gutkin, P. Braun, M. Akila, D. Waltner, and T. Guhr, Local correlations in dual-unitary kicked chains, arXiv:2001.01298 [cond-mat, physics:math-ph, physics:nlin] (2020).
  * Claeys and Lamacraft [2021] P. W. Claeys and A. Lamacraft, Ergodic and Nonergodic Dual-Unitary Quantum Circuits with Arbitrary Local Hilbert Space Dimension, Physical Review Letters 126, 100603 (2021).
  * Bertini  _et al._ [2019a] B. Bertini, P. Kos, and T. Prosen, Exact Correlation Functions for Dual-Unitary Lattice Models in $1+1$ Dimensions, Phys. Rev. Lett. 123, 210601 (2019a).
  * Bertini  _et al._ [2019b] B. Bertini, P. Kos, and T. Prosen, Entanglement Spreading in a Minimal Model of Maximal Many-Body Quantum Chaos, Physical Review X 9, 021033 (2019b).
  * Bertini and Piroli [2020] B. Bertini and L. Piroli, Scrambling in random unitary circuits: Exact results, Physical Review B 102, 064305 (2020).
  * Kos  _et al._ [2021] P. Kos, B. Bertini, and T. Prosen, Correlations in Perturbed Dual-Unitary Circuits: Efficient Path-Integral Formula, Physical Review X 11, 011022 (2021).
  * Piroli  _et al._ [2020] L. Piroli, B. Bertini, J. I. Cirac, and T. Prosen, Exact dynamics in dual-unitary quantum circuits, Physical Review B 101, 094304 (2020).
  * Bertini  _et al._ [2021] B. Bertini, P. Kos, and T. Prosen, Random Matrix Spectral Form Factor of Dual-Unitary Quantum Circuits, Communications in Mathematical Physics 387, 597 (2021).
  * Gullans and Huse [2020b] M. J. Gullans and D. A. Huse, Scalable Probes of Measurement-Induced Criticality, Physical Review Letters 125, 070606 (2020b).
  * Bao  _et al._ [2020] Y. Bao, S. Choi, and E. Altman, Theory of the phase transition in random unitary circuits with measurements, Physical Review B 101, 104301 (2020).
  * Choi  _et al._ [2020] S. Choi, Y. Bao, X.-L. Qi, and E. Altman, Quantum Error Correction in Scrambling Dynamics and Measurement-Induced Phase Transition, Physical Review Letters 125, 030505 (2020).
  * Jian  _et al._ [2020] C.-M. Jian, Y.-Z. You, R. Vasseur, and A. W. W. Ludwig, Measurement-induced criticality in random quantum circuits, Physical Review B 101, 104302 (2020).
  * Ippoliti  _et al._ [2021] M. Ippoliti, M. J. Gullans, S. Gopalakrishnan, D. A. Huse, and V. Khemani, Entanglement Phase Transitions in Measurement-Only Dynamics, Physical Review X 11, 011030 (2021).
  * Lavasani  _et al._ [2021a] A. Lavasani, Y. Alavirad, and M. Barkeshli, Measurement-induced topological entanglement transitions in symmetric random quantum circuits, Nature Physics 17, 342 (2021a).
  * Fidkowski  _et al._ [2021] L. Fidkowski, J. Haah, and M. B. Hastings, How Dynamical Quantum Memories Forget, Quantum 5, 382 (2021).
  * Fan  _et al._ [2021] R. Fan, S. Vijay, A. Vishwanath, and Y.-Z. You, Self-organized error correction in random unitary circuits with measurement, Phys. Rev. B 103, 174309 (2021).
  * Lavasani  _et al._ [2021b] A. Lavasani, Y. Alavirad, and M. Barkeshli, Topological Order and Criticality in $(2+1)\mathrm{D}$ Monitored Random Quantum Circuits, Physical Review Letters 127, 235701 (2021b).
  * Szyniszewski  _et al._ [2020] M. Szyniszewski, A. Romito, and H. Schomerus, Universality of Entanglement Transitions from Stroboscopic to Continuous Measurements, Phys. Rev. Lett. 125, 210602 (2020).
  * Nahum and Skinner [2020] A. Nahum and B. Skinner, Entanglement and dynamics of diffusion-annihilation processes with Majorana defects, Physical Review Research 2, 023288 (2020).
  * Nahum  _et al._ [2021] A. Nahum, S. Roy, B. Skinner, and J. Ruhman, Measurement and Entanglement Phase Transitions in All-To-All Quantum Circuits, on Quantum Trees, and in Landau-Ginsburg Theory, PRX Quantum 2, 010352 (2021).
  * Li and Fisher [2021] Y. Li and M. P. A. Fisher, Statistical mechanics of quantum error correcting codes, Physical Review B 103, 104306 (2021).
  * Li  _et al._ [2021] Y. Li, S. Vijay, and M. P. A. Fisher, Entanglement Domain Walls in Monitored Quantum Circuits and the Directed Polymer in a Random Environment, arXiv:2105.13352 [cond-mat, physics:quant-ph] (2021).
  * Agrawal  _et al._ [2022] U. Agrawal, A. Zabalo, K. Chen, J. H. Wilson, A. C. Potter, J. H. Pixley, S. Gopalakrishnan, and R. Vasseur, Entanglement and Charge-Sharpening Transitions in U(1) Symmetric Monitored Quantum Circuits, Physical Review X 12, 041002 (2022).
  * Potter and Vasseur [2022] A. C. Potter and R. Vasseur, Entanglement Dynamics in Hybrid Quantum Circuits, in  _Entanglement in Spin Chains: From Theory to Quantum Technology Applications_, Quantum Science and Technology, edited by A. Bayat, S. Bose, and H. Johannesson (Cham, 2022) pp. 211–249.
  * Barenco  _et al._ [1995] A. Barenco, C. H. Bennett, R. Cleve, D. P. DiVincenzo, N. Margolus, P. Shor, T. Sleator, J. A. Smolin, _et al._ , Elementary gates for quantum computation, Physical Review A 52, 3457 (1995).
  * Marvian and Spekkens [2014] I. Marvian and R. W. Spekkens, A Generalization of Schur-Weyl Duality with Applications in Quantum Estimation, Communications in Mathematical Physics 331, 431 (2014).
  * Swingle  _et al._ [2016] B. Swingle, G. Bentsen, M. Schleier-Smith, and P. Hayden, Measuring the scrambling of quantum information, Phys. Rev. A 94, 040302 (2016).
  * Sunderhauf  _et al._ [2019] C. Sunderhauf, L. Piroli, X.-L. Qi, N. Schuch, and J. I. Cirac, Quantum chaos in the Brownian SYK model with large finite N : OTOCs and tripartite information, Journal of High Energy Physics 2019, 38 (2019).
  * Gharibyan  _et al._ [2018] H. Gharibyan, M. Hanada, S. H. Shenker, and M. Tezuka, Onset of random matrix behavior in scrambling systems, Journal of High Energy Physics 2018, 124 (2018).
  * Alba and Calabrese [2017] V. Alba and P. Calabrese, Entanglement and thermodynamics after a quantum quench in integrable systems, Proceedings of the National Academy of Sciences 114, 7947 (2017).
  * Calabrese [2020] P. Calabrese, Entanglement spreading in non-equilibrium integrable systems, SciPost Physics Lecture Notes , 20 (2020).
  * Gopalakrishnan [2018] S. Gopalakrishnan, Operator growth and eigenstate entanglement in an interacting integrable Floquet system, Physical Review B 98, 060302 (2018).
  * Klobas  _et al._ [2021] K. Klobas, B. Bertini, and L. Piroli, Exact Thermalization Dynamics in the “Rule 54” Quantum Cellular Automaton, Physical Review Letters 126, 160602 (2021).
  * Buca  _et al._ [2021] B. Buca, K. Klobas, and T. Prosen, Rule 54: exactly solvable model of nonequilibrium statistical mechanics, Journal of Statistical Mechanics: Theory and Experiment 2021, 074001 (2021).
  * Bao  _et al._ [2022] Y. Bao, M. Block, and E. Altman, Finite time teleportation phase transition in random quantum circuits, arXiv:2110.06963 [cond-mat, physics:quant-ph] (2022).
  * Napp  _et al._ [2022] J. C. Napp, R. L. La Placa, A. M. Dalzell, F. G. S. L. Brandao, and A. W. Harrow, Efficient Classical Simulation of Random Shallow 2D Quantum Circuits, Physical Review X 12, 021021 (2022).
  * Ippoliti and Ho [2022] M. Ippoliti and W. W. Ho, Solvable model of deep thermalization with distinct design times, Quantum 6, 886 (2022).
  * Zhou and Nahum [2019] T. Zhou and A. Nahum, Emergent statistical mechanics of entanglement in random unitary circuits, Phys. Rev. B 99, 174205 (2019).
  * Zhou and Nahum [2020] T. Zhou and A. Nahum, Entanglement Membrane in Chaotic Many-Body Systems, Phys. Rev. X 10, 031066 (2020).
  * Vasseur  _et al._ [2019] R. Vasseur, A. C. Potter, Y.-Z. You, and A. W. W. Ludwig, Entanglement transitions from holographic random tensor networks, Physical Review B 100, 134203 (2019).
  * Weinstein  _et al._ [2022] Z. Weinstein, Y. Bao, and E. Altman, Measurement-Induced Power-Law Negativity in an Open Monitored Quantum Circuit, Physical Review Letters 129, 080501 (2022).
  * Foss-Feig  _et al._ [2021] M. Foss-Feig, D. Hayes, J. M. Dreiling, C. Figgatt, J. P. Gaebler, S. A. Moses, J. M. Pino, and A. C. Potter, Holographic quantum algorithms for simulating correlated spin systems, Phys. Rev. Research 3, 033002 (2021).