# Bridging the Sim2Real gap with CARE:
Supervised Detection Adaptation with Conditional Alignment and Reweighting

Viraj Prabhu  David Acuna  Andrew Liao  Rafid Mahmood  Marc T. Law  Judy
Hoffman  Sanja Fidler  James Lucas

###### Abstract

Sim2Real domain adaptation (DA) research focuses on the constrained setting of
adapting from a labeled synthetic source domain to an unlabeled or sparsely
labeled real target domain. However, for high-stakes applications (_e.g._
autonomous driving), it is common to have a modest amount of human-labeled
real data in addition to plentiful auto-labeled source data (_e.g._ from a
driving simulator). We study this setting of _supervised_ sim2real DA applied
to 2D object detection. We propose Domain Translation via Conditional
Alignment and Reweighting (CARE) a novel algorithm that systematically
exploits target labels to explicitly close the sim2real appearance and content
gaps. We present an analytical justification of our algorithm and demonstrate
strong gains over competing methods on standard benchmarks.

Machine Learning, Domain Adaptation, Computer Vision

##  1 Introduction

Domain Adaptation (DA) is a framework that seeks to overcome shifts in data
distributions between training and testing. Typically, DA methods assume
access to a large amount of labeled data from the training (source)
distribution, and unlabeled or sparingly labeled data from the test (target)
distribution (Saenko et al., 2010; Ganin & Lempitsky, 2015; Tzeng et al.,
2017). DA has been extensively studied in computer vision for applications
where annotating target data is expensive (Csurka et al., 2022).

As annotation costs decrease, it becomes increasingly practical to annotate
more target data, especially in high-stakes industrial applications such as
autonomous driving (Mahmood et al., 2022). A common practice in this field is
to augment a target dataset of real driving scenarios with an additional
labeled dataset generated in simulation (Karpathy, 2021; Kishore et al.,
2021). Simulated data may be particularly useful to improve performance on the
long-tail of driving scenarios for which it may be difficult to challenging to
collect real labeled data (Rempe et al., 2022; Resnick et al., 2022).In this
paper, we formulate this setting as a _supervised_ Sim2Real DA problem. We use
simulated, machine-labeled source data, and real, human-labeled target data
(see Fig. 1), and ask: in this label-privileged setting, what would be the
most effective way to combine sim and real data to improve target performance?

Figure 1:  Traditional Sim2Real domain adaptation assumes access to very few
or no target labels, which is unrealistic in high-stakes applications like
self-driving. We study the practical setting of _supervised_ Sim2Real domain
adaptation applied to 2D object detection, wherein the goal is to maximize
heldout target performance given access to human-labeled target data and an
additional large set of machine-labeled simulated data.

Table 1:  Car detection adaptation from Sim10K$\to$Cityscapes: Systematically
combining labeled source and target data improves over using a single data
source as well as naïve combinations.

Method | mAP@50 ($\uparrow$)
---|---
Only labeled source data | 41.8
Only labeled target data | 62.1
UDA (Khindkar et al., 2022) (Labeled source $+$ unlabeled target) | 53.1
FDA (Wang et al., 2020) (Labeled source $+$ labeled target) | 65.2
Mixing (Kishore et al., 2021) (Labeled Source $+$ labeled target) | 64.8
Seq. FT (Tremblay et al., 2018) (Labeled source $+$ labeled target) | 66.4
Ours - CARE (Labeled source $+$ labeled target) | 68.1

Surprisingly, this practical setting has received little interest in recent
domain adaptation literature, which focuses on unsupervised adaptation  _(no
target labels,_ Chen et al. (2018); Acuna et al. (2021a); Li et al. (2022)),
and few-shot and semi-supervised adaptation (_few target labels_ , Donahue et
al. (2013); Wang et al. (2019); Saito et al. (2019a); Wang et al. (2020)).
Although such methods could be extended to the supervised setting, _e.g._ by
adding a supervised target loss to an off-the-shelf unsupervised DA method, we
find this to be suboptimal in practice (see Table 1), since these
straightforward extensions do not exploit large-scale target labels and their
statistics for domain alignment. Similarly, few-shot and semi-supervised
adaptation methods assume access to limited target labels (_e.g._ 8 labeled
images per class for object detection, Wang et al. (2019)) that are
insufficient for reliably estimating target statistics. Facing this research
gap, industry practitioners may resort to naïvely combining labeled source and
target data via mixing (Kishore et al., 2021) (_i.e._ training on combined
source and target data) or sequential fine-tuning (Tremblay et al., 2018;
Prakash et al., 2019, 2021) (_i.e._ training on source data followed by
finetuning on target data). However, these simple heuristics do not address
the domain gap between simulation and reality.

This paper addresses the research-practice gap to show that _systematically_
combining the two labeled data sets can significantly improve performance over
competing methods (see Table 1). We propose a general framework called _Domain
Translation via Conditional Alignment and Reweighting_ (CARE) for supervised
Sim2Real DA. CARE builds on commonly-used baselines and off-the-shelf
adaptation methods but explicitly leverages existing labels in the target
domain to minimize both appearance (pixel and instance-level visual disparity)
and content gaps (disparities in task label distributions and scene layout).
Specifically, we overcome the appearance gap by explicitly using ground-truth
labels to conditionally align intermediate instance representations. To
overcome the content gap, we conditionally reweight the importance of samples
using estimated spatial, size, and categorical distributions. We formalize our
setting using the joint risk minimization framework, and provide theoretical
insights for our design choices. Finally, we apply our framework to the
challenging task of 2D object detection We make the following contributions:

(1) We present a detailed study of supervised Sim2Real object detection
adaptation and show that existing methods yield suboptimal performance by not
adequately exploiting target labels. (2) We propose CARE, a general framework
for supervised Sim2Real domain adaptation and apply it to the 2D object
detection. On three standard Sim2Real benchmarks for detection adaptation,
CARE strongly outperforms competing methods (_e.g._ boosting mAP@50 by as much
as $\sim$25% on Synscapes$\to$Cityscapes). (3) We formalize our setting using
the joint risk minimization framework and provide theoretical insights into
our design choices.

##  2 Related work

To our knowledge, supervised domain adaptation (SDA) for object detection has
not seen recent work in computer vision. Early DA works (Saenko et al., 2010;
Kulis et al., 2011; Hoffman et al., 2013; Tsai et al., 2016) have studied the
SDA setting applied to image classification, proposing contrastive-style
approaches based on metric learning with cross-domain pairwise constraints.
However, these works predate deep learning and do not study complex tasks like
object detection. Below, we summarize lines of work in the related areas of
unsupervised and few-shot adaptation.

Unsupervised domain adaptation (UDA). The DA literature primarily focuses on
_unsupervised_ adaptation from a labeled source setting to an unlabeled target
domain (Saenko et al., 2010; Ganin & Lempitsky, 2015; Hoffman et al., 2018).
Successful UDA approaches have employed different strategies ranging from
domain adversarial learning (Long et al., 2015; Acuna et al., 2021b) to domain
discrepancy minimization (Long et al., 2018), image translation (Hoffman et
al., 2018), and self-training (Prabhu et al., 2021; Li et al., 2022). Cross-
domain object detection has also seen recent work, based on multi-level domain
adversarial learning (Chen et al., 2018), strong-weak distribution alignment
of local and global features (Saito et al., 2019b), and domain adversarial
learning weighted by region discriminativeness (Zhu et al., 2019),
Alternatively, RoyChowdhury et al. (2019); Li et al. (2022) self-train with
refined pseudolabels, and Kim et al. (2019) use background regularization.

Importantly, due to the absence of target labels, UDA methods resort to
approximations based on marginal alignment or pseudolabels. In this paper, we
instead consider _supervised_ Sim2Real adaptation where ground-truth labels
are provided for the target dataset during training. To compare against our
approach, we benchmark supervised extensions of existing UDA methods as
baselines in our paper.

Figure 2:  The domain gap between a simulated source and real target domain
consists of an appearance and content gap. The appearance gap corresponds to
pixel-level differences (_e.g._ texture and lighting) and instance-level
differences (_e.g._ vehicle design). The content gap consists of differences
in label distributions due to different class frequencies and bounding box
sizes and locations. Right. Column 1: Task label histograms. Column 2:
Empirical distribution of “car” box _sizes_. Column 3: Empirical distribution
of “car” box _locations_.

Few-shot (FDA) and Semi-supervised Domain Adaptation (SSDA). Closer to our
setting are Few-shot DA learning (FDA, Wang et al. (2019); Gao et al. (2022);
Zhong et al. (2022); Ramamonjison et al. (2021)) and Semi-supervised DA(SSDA,
Donahue et al. (2013); Yao et al. (2015); Saito et al. (2019a)), which differ
in important ways. FDA assumes a very small amount of labeled target data is
available (_e.g._ 8 images per-class for detection in Wang et al. (2019)).
Such methods employ source feature-regularized images with instance-level
adversarial learning (Wang et al., 2019), point-wise distribution alignment
(Zhong et al., 2022), and multi-level domain-aware data augmentation (Gao et
al., 2022). SSDA also assumes limited target labels (_e.g._ 1 to 3 images per
category for image classification (Saito et al., 2019a)), but additionally
leverages a large set of _unlabeled_ target data, making use of min-max
entropy optimization (Saito et al., 2019a) or student-teacher learning
frameworks (Li et al., 2022). Instead, we operate in a _supervised_ DA setting
with access to a substantial amount of labeled target data in addition to a
large (in theory, possibly infinite) amount of labeled simulated data. As a
result, SDA uniquely permits _reliable_ estimates of target statistics. Our
algorithm leverages these statistics and target labels to systematically close
the Sim2Real domain gap.

##  3 Approach

In this section, we first introduce the supervised Sim2Real detection
adaptation problem (Section 3.1). We characterize two primary aspects of the
Sim2Real domain gap: an appearance and a content gap (Section 3.2). Finally we
introduce our method CARE that leverages a labeled target dataset to close
this domain gap (Section 3.3) and provide an analytical justification of the
algorithm (Section 3.4).

###  3.1 Problem Formulation

Let $\mathcal{X}$ and $\mathcal{Y}$ denote input and output spaces. In object
detection, $x\in\mathcal{X}$ are images
($\mathcal{X}\subseteq\mathbb{R}^{H\times W\times 3}$) and
$y:=(B,C)\in\mathcal{Y}$ are $K$-class labels with $C\in\\{1,..,K\\}$ and
bounding boxes
$B\subseteq\\{(\mathsf{w},\mathsf{h},\mathsf{x},\mathsf{y})\in\mathbb{R}^{4}\\}$
(comprising the width $\mathsf{w}$, height $\mathsf{h}$, and centre
coordinates $(\mathsf{x},\mathsf{y})$, respectively). Let
$h(x):=h_{\theta}(g_{\phi}(x))$ be an object detector composed of a feature
extractor $g(x)$ and a classifier $h(g(x))$ that are parameterized by $\phi$
and $\theta$. Matching prior object detection work (Khindkar et al., 2022;
Wang et al., 2021), we design $h(g(x))$ via Faster RCNN (Ren et al., 2015),
which uses a region proposal network that receives features generated by a
backbone network and passes them through an ROI align layer to obtain ROI
features; these are then passed through a final box predictor. We let
$\hat{B},\hat{C}=\arg\max h(g(x))$ be bounding box coordinates and object
class predicted by the model for input image $x$. In sim2real SDA, we are
given two labeled data sets representing a (simulated) source distribution
$P_{S}$ and a (real) target distribution $P_{T}$. Our goal is to minimize the
expected risk of a detection loss consisting of a classification loss
$\ell_{cls}$ and bounding box regression loss $\ell_{box}$:

$\ell_{det}(h(g(x)),B,C):=\ell_{box}(\hat{B},B)+\ell_{cls}(\hat{C},C)$ (1)

over a target domain $r_{T}:=\mathbb{E}_{x,B,C\sim
P_{T}}[\ell_{det}(h(x),B,C)]$.

###  3.2 Characterizing the Sim2Real Domain Gap

Leveraging the source distribution to improve performance on the target is
challenging due to the _domain gap_ which exists in both the image and label
distributions. We partition this gap into two categories: appearance and
content gap (Kar et al., 2019) and characterize these in detail, using the
Synscapes (Wrenninge & Unger, 2018)$\to$ Cityscapes (Cordts et al., 2016)
shift for object detection adaptation as an example.

The appearance gap consists of visual disparities between images from the two
domains (see Fig 2, _left_). For example, a pixel-level appearance gap may be
due to differences in lighting between real and simulated images
(Chattopadhyay et al., 2022), while an instance-level gap may be due to
differences in the appearance of synthesized versus real objects. We
characterize the appearance gap as the dissimilarity $D(\cdot,\cdot)$ in the
probabilities between source and target distributions when conditioned on the
label (e.g. $D(P_{S}(x|B,C),P_{T}(x|B,C))$).

The content gap can be decomposed into scene-level changes in the layout of
objects (e.g., size and spatial distribution) as well as shifts in the task
label distributions and the frequencies of classes (see Fig 2, _right_). We
characterize the scene-level changes as the dissimilarity in the probabilities
of object bounding boxes when conditioned on the class
$D(P_{S}(B|C),P_{T}(B|C))$ and the task-level class frequency gap as the
dissimilarity in class probabilities $D(P_{S}(C),P_{T}(C))$.

Figure 3:  Conditional Alignment and Reweighting (CARE) exploits target labels
to estimate and bridge cross-domain appearance gaps (via a cycle consistency-
based conditional feature alignment objective) and content gaps (via
importance reweighting).

###  3.3 Bridging the domain gap with CARE

To close the sim2real gap, _Conditional Alignment and Reweighting_ (CARE)
minimizes the effect of both the appearance and the content gap via feature
alignment and importance reweighing. Let
$w_{S}(C):=1/P_{S}(C),w_{T}(C):=1/P_{T}(C)$ be the inverse class frequency for
each domain and let $v(B|C):=P_{T}(B|C)/P_{S}(B|C)$ be the inverse ratio of
the scene-level bounding box frequency gap. These reweighting factors ensure
that the learned classifier considers that the source and target data sets
follow the same distribution during training. In CARE, we minimize the
following _domain translation_ loss:

$\displaystyle\begin{split}\min_{\theta,\phi}&\ \mathbb{E}_{x,B,C\sim
P_{S}}\bigg{[}w_{S}(C)v(B|C)\ell_{det}(h(g(x)),B,C)\bigg{]}\\\
&+\mathbb{E}_{x^{\prime},B^{\prime},C^{\prime}\sim
P_{T}}\big{[}w_{T}(C^{\prime})\ell_{det}(h(g(x^{\prime})),B^{\prime},C^{\prime})\big{]}\\\
&+\lambda\mathbb{E}_{\genfrac{}{}{0.0pt}{}{x^{\prime},B^{\prime},C^{\prime}\sim
P_{T}}{x,B,C\sim
P_{S}}}\bigg{[}\ell_{align}(g(x),g(x^{\prime}))\bigg{|}C=C^{\prime}\bigg{]}.\end{split}$
(2)

where $\ell_{align}$ is defined in Eq. (3), and $\lambda\geq 0$ is a
regularization parameter. The above loss minimizes three terms, where the
first term is a reweighted detection loss over the source dataset and the
second loss is a class-balanced detection loss over the target dataset. The
third term aligns the encoded features $g(x)$ and $g(x^{\prime})$ of similar
cross-domain instance embeddings belonging the same class. We now elaborate
upon each term.

####  3.3.1 Bridging appearance gap with cross-domain cycle consistency

To minimize the appearance gap, $\ell_{align}$ performs a class-and-box
conditional feature alignment strategy by optimizing a cross-domain cycle
consistency objective. Specifically, we extract ROI features corresponding to
the ground truth bounding box coordinates of both source and target images and
match similar cross-domain instance features belonging to the same class. Fig.
4 visualizes the intuition.

For a given class, suppose we are given $k$ ground truth bounding boxes from
the source and target domains each. For each instance, our encoder extracts
$d$-dimensional ROI features $\mathbf{f}^{i}_{\omega}\in\mathbb{R}^{d}$, where
$i\in\\{1,\dots,k\\}$ and $\omega\in\\{S,T\\}$ denote the $i$-th feature and
the domain, respectively. We first measure the (negative of the) squared
Euclidean distance between these same-class cross-domain features:

$s_{i,j}:=-\|\mathbf{f}_{S}^{i}-\mathbf{f}_{T}^{j}\|_{2}^{2}.$

For each target $j$, we compute soft-matching features

$\hat{\mathbf{f}}_{T}^{j}:=\sum_{j^{\prime}=1}^{k}\alpha_{j,j^{\prime}}\mathbf{f}_{T}^{j^{\prime}},~{}\text{where
}~{}\alpha_{j,j^{\prime}}:=\frac{e^{s_{j,j^{\prime}}}}{\sum_{m=1}^{k}e^{s_{j,m}}}$

Finally, we assemble a similarity score between each source $i$ and target $j$
instance by minimizing the negative squared Euclidean distance between the
source and the soft-matching target feature vectors

$\hat{s}_{i,j}:=-\|\mathbf{f}_{S}^{i}-\hat{\mathbf{f}}_{T}^{j}\|_{2}^{2}.$

Let $\mathbf{\hat{s}}^{j}:=\left[\hat{s}_{1,j},\dots,\hat{s}_{k,j}\right]$ be
the vector of similarity scores for the $j$-th target. Our cycle matching
alignment loss minimizes the cross entropy between features as follows:

(3)

Figure 4:  Visualization of cross-domain cycle consistency matching with CARE
on Sim10K$\to$Cityscapes. CARE embeds similar-looking cars closer to minimize
the appearance gap.

The above approach is a modification of a temporal cycle confusion objective
proposed for robust object detection (Wang et al., 2021). However, we differ
in three ways. First, we align cross-domain instance features between source
and target domains, whereas the original approach aligns instance features
across time given video data. Second, we leverage target labels to align ROI
features corresponding to _ground truth_ rather than predicted bounding box
coordinates. Finally, our alignment objective uses cycle _consistency_ rather
than cycle confusion. Intuitively, we encourage _similar-looking_ instances to
be close together (by taking the negative Euclidean distance), whereas the
original aligns dissimilar instances. Our alignment loss reduces to the
classification of the soft nearest neighbors and therefore tends to be robust
to label noise (Dwibedi et al., 2019).

####  3.3.2 Bridging content gap with importance reweighting

To close the task label distribution content gap, we apply inverse frequency
reweighing to simulate a balanced label distribution in the source and target
domains. For each domain $\omega\in\\{S,T\\}$, we reweigh instances of class
$C$ via multiplicative class weights $w_{\omega}(C)\propto 1/N_{\omega}(C)$,
where $N_{\omega}(C)$ is the number of training examples in domain $\omega$.

We approximate the class-conditional box ratios as follows

$\displaystyle\frac{P_{T}(B|C)}{P_{S}(B|C)}\approx\frac{P_{T}(\mathsf{w},\mathsf{h}|C)}{P_{S}(\mathsf{w},\mathsf{h}|C)}\frac{P_{T}(\mathsf{x},\mathsf{y}|C)}{P_{S}(\mathsf{x},\mathsf{y}|C)}=:v(B|C)$
(4)

Intuitively, this ratio upweighs boxes of a class that are of a size and
location relatively more represented in the target than in the source. Note
that the first approximate equality $\approx$ is due to an assumption of
independence between $(\mathsf{w},\mathsf{h})$ and $(\mathsf{x},\mathsf{y})$,
which we assume to simplify computations. We estimate each probability
component via class-conditional Gaussian kernel density estimation (KDE)
(Scott, 2015) fitted to the ground truth bounding box locations and sizes
respectively. In Appendix A.2, we include details of this estimation,
including appropriate smoothing and thresholding to handle regions with low
target support.

###  3.4 Analytical justification

We now analyze our loss function in Eq. (2) to develop a theoretical intuition
for its effectiveness. Let us rewrite the first term in the loss as follows:

$\displaystyle\mathbb{E}_{P_{S}}\bigg{[}w_{S}(C)v(B|C)\ell_{det}(h(g(x)),B,C)\bigg{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{P_{T}}\bigg{[}\frac{P_{S}(x,B,C)}{P_{T}(x,B,C)}w_{S}(C)v(B|C)\ell_{det}(h(g(x)),B,C)\bigg{]}$
$\displaystyle\begin{split}=&\mathbb{E}_{P_{T}}\bigg{[}\frac{P_{S}(C)}{P_{T}(C)}\times\frac{P_{S}(B|C)}{P_{T}(B|C)}\times\frac{P_{S}(x|B,C)}{P_{T}(x|B,C)}\\\
&\qquad\quad\times w_{S}(C)v(B|C)\ell_{det}(h(g(x)),B,C)\bigg{]}.\end{split}$
(5)

Above, the second line follows from importance reweighting, and the third line
follows from Bayes rule. Next, recall that $w_{S}(C)=1/P_{S}(C)$ and
$v(B|C)\approx P_{T}(B|C)/P_{S}(B|C)$. Substituting these two, we obtain

(6)

Finally, recall our feature alignment component, which is designed to minimize
the distance between encoded features of the same class and box statistics.
Successfully minimizing the third term in Eq. (2) should obtain
$P_{S}(g(x)|B,C)=P_{T}(g(x)|B,C)$. Using this, we obtain

$\displaystyle\text{Eq. }\eqref{eq:simplified_first_term}$
$\displaystyle\approx\mathbb{E}_{P_{T}}\bigg{[}\frac{P_{S}(g(x)|B,C)}{P_{T}(g(x)|B,C)}\frac{1}{P_{T}(C)}\ell_{det}(h(g(x)),B,C)\bigg{]}$
$\displaystyle=\mathbb{E}_{P_{T}}\bigg{[}\frac{1}{P_{T}(C)}\ell_{det}(h(g(x)),B,C)\bigg{]}$
(7)

where the first line follows from the assumption that feature-level distances
should reflect image appearance distances, and the second line follows from
minimizing $\ell_{align}$. Overall, Eq. (7) and the second term in Eq. (2)
minimize a class-weighted version of the expected risk $r_{T}$. In our case,
the target metric is mean AP, which values performance on all classes equally.
Since in practice, our target data distributions often feature imbalanced
classes, this modified risk simulates a balanced class label distribution and
better maximizes mAP.

We remark that the steps here follow from several assumptions including
independence of box position and size, equivalence between the ratios of
feature-level probabilities and appearance probabilities, and that the target
support is a subset of that of the source. Further, it relies on successfully
minimizing this feature-level gap. Nonetheless as we show in the next section,
our method demonstrates powerful empirical performance in the target domain.

##  4 Results

We now describe our experimental setup for object detection adaptation:
datasets and metrics (Section 4.1), implementation details (Section 4.2), and
baselines (Section 4.3). We then present our results (Section 4.4) and ablate
(Section 4.5) and analyze our approach (Section 4.6).

###  4.1 Datasets and metrics

Table 2: Results for supervised sim2real object detection adaptation on
target. We compare CARE to source and target only training, a state-of-the-art
unsupervised DA method (ILLUME (Khindkar et al., 2022)), naive sim+real
combinations (mixing (Kishore et al., 2021) and sequential finetuning
(Tremblay et al., 2018)), supervised extensions of popular UDA methods (DANN
(Ganin & Lempitsky, 2015) and MMD (Long et al., 2015)),and a recently proposed
few-shot detection strategy (Wang et al., 2020).

[][][b]  Method mAP@50 ($\uparrow$) Source 41.8 UDA 53.1 Target 62.1 Mixing
64.8 Seq. FT 66.4 S-MMD 65.8 S-DANN 65.3 FDA 65.2 CARE (ours) 68.1

Table 3: Sim10K$\to$Cityscapes (1 class)

[][][!htbp]  Method mAP@50 ($\uparrow$) Source 19.2 Target 34.2 Mixing 39.0
Seq. FT 39.8 S-MMD 40.0 S-DANN 40.8 CARE (ours) 48.5

Table 4: Synscapes$\to$Cityscapes (8 classes)

[][][!htbp]  Method mAP@50 ($\uparrow$) Source 22.5 Target 45.2 Mixing 49.3
Seq. FT 45.4 S-MMD 50.6 S-DANN 49.8 CARE (ours) 53.7

Table 5: DriveSim$\to$CityScapes (3 classes)

We perform domain adaptation from three different source data sets of
synthetic images, Sim10K (Johnson-Roberson et al., 2017), Synscapes (Wrenninge
& Unger, 2018), and DriveSim, an internal data set simulated using an early
version of NVIDIA DriveSim (NVIDIA, 2021), following the procedure described
in Acuna et al. (2021a). Sim10K contains 10,000 images of 1914$\times$1052
resolution with pixel-level annotations extracted from the game GTA-5.
Synscapes is a photorealistic dataset of 25,000 synthetic driving scenes of
1440$\times$720 resolution. Finally, DriveSim is a private synthetic data set
of 48,000 photorealistic driving scenes. Synscapes and DriveSim exhibit a
long-tailed category distribution (see Fig. 2). For each source, we train an
object detector to adapt to our target, Cityscapes (Cordts et al., 2016) which
is a data set of 2500 real driving images. For all evaluations, we fix the
target data set size to 25% to model the realistic scenario of available but
an order of magnitude less real data than synthetic data (see appendix for
details). For Sim10K$\to$Cityscapes, we focus on object detection for a single
class (_i.e._ car) to better compare against prior Sim2Real domain adaptation
methods (Khindkar et al., 2022). For Synscapes$\to$Cityscapes and
DriveSim$\to$CityScapes, we evaluate object detection for eight and three
classes, respectively. To evaluate all models, we match prior work (Chen et
al., 2018; Khindkar et al., 2022; Wang et al., 2021) and report per-category
Average Precision (AP) and its mean across classes at an IoU threshold of 50%
(mAP@50), over the target test set.

###  4.2 Implementation details

We use a Faster-RCNN (Ren et al., 2015) architecture with a ResNet-50 (He et
al., 2016) backbone. We run 10k iterations of SGD with a learning rate of
0.01, momentum of 0.9, weight decay of 10-4, and learning rate warmup matching
(Wang et al., 2021). We set $\lambda=0.1$ in Eq. (2). We use 8 NVIDIA V100
GPUs with a per-GPU batch size of 4, and maintain a 1:1 within-batch source to
target ratio across experiments.

###  4.3 Baselines

We compare against: (1) Source only: Supervised learning using only the
labeled source dataset. (2) Target only: Supervised learning using only the
labeled target dataset. (3) Mixing (Kishore et al., 2021): Supervised learning
on the combined source and target data sets, while maintaining a 1:1 ratio
within batches (we ablate this mixing ratio in appendix). (4) Sequential
Finetuning (Tremblay et al., 2018): Supervised learning on the source dataset
followed by finetuning all layers of the model with the target dataset. (5)
Unsupervised DA (UDA) with ILLUME (Khindkar et al., 2022): For completeness,
we copy results on Sim10K$\to$Cityscapes of a state-of-the-art UDA method that
uses labeled source and unlabeled target data.

We also propose and benchmark supervised extensions of two popular UDA
strategies: (6) S-MMD: A class and box-conditional _supervised_ version of
Maximum Mean Discrepancy (Long et al., 2015). S-MMD minimizes the MMD loss
between cross-domain box features corresponding to the same class, using a
linear kernel. (7) S-DANN: A class and box-conditional _supervised_ version of
DANN (Ganin & Lempitsky, 2015). S-DANN minimizes the domain adversarial loss
between cross-domain box features corresponding to the same class, similar to
Chen et al. (2018). (8) Few-shot DA (FDA) with TFA: (Wang et al., 2020). This
is a two-stage finetuning algorithm proposed for few-shot object detection
that updates all parameters on source (base) data followed by finetuning only
the final layer (box regressor and classifer) on a balanced dataset of source
and target data. However, we observe low performance with finetuning only the
last layer (despite using a lower learning rate as recommended and both with
and without weight re-initialization). Instead, we report results _without_
freezing weights in the second phase.

###  4.4 Main Results

Table 5 summarizes our results. We find:

Table 6:  Ablating our proposed method on all three shifts. Our method is in
gray with the improvement versus mixing in small font.

# | $P(g(x)|B,C)$ | $P(C)$ | $P(B|C)$ |  mAP@50 ($\uparrow$)
---|---|---|---|---
alignment | rewt. | rewt. | Sim10k | Synscapes | DriveSim
1 | (Mixing baseline) | 64.8 | 39.0 | 49.3
2 | S-MMD |  |  | 65.8 | 40.0 | 50.6
3 | S-DANN |  |  | 65.3 | 40.8 | 49.8
4 | Cycle Consistency |  |  | 67.2 | 41.8 | 50.8
5 | None (Mixing baseline) | ✓ |  | 64.8 | 46.1 | 51.8
6 | Cycle Consistency | ✓ |  | 67.2 | 46.6 | 52.5
7 | Cycle Consistency | ✓ | ✓ | 68.1+3.3 | 48.5+9.5 | 53.7+4.4

$\triangleright$ Simulated data and labeled real data are both needed. We
first confirm that supervised learning using only the target data outperforms
both the settings of using only source data and unsupervised domain adaptation
with unlabeled target data. Moreover across all three shifts, even baselines
that naïvely combine simulated and real data (_i.e._ mixing and sequential
finetuning) outperform training using only the target data. This shows that
additional simulated data is helpful. Moreover, sequential finetuning
outperforms mixing on two of three shifts. Finally, we find that mixing with
additional conditional feature alignment (S-MMD, S-DANN), consistently
outperforms naive mixing. Additional results are in Appendix A.1.

$\triangleright$ CARE outperforms all competing methods. First, note that
across each shift, CARE outperforms mixing (+3.3, +9.5, +4.4 mAP@50) and
sequential finetuning (+1.7, +8.7, +8.3 mAP@50). This suggests that the
Sim2Real domain gap is a barrier to effective mixing, and systematically
mitigating it using target labels is beneficial. Most importantly, we
outperform each benchmarked supervised extension of UDA on all shifts. This
result validates the research-practice gap by showing that UDA cannot be
easily extended to the practical setting of labeled target data, thereby
necessitating CARE in supervised domain adaptation.

###  4.5 Ablation study

In Table 6, we ablate the various components of CARE.

$\triangleright$ Class-and-box conditional feature alignment is necessary
(Rows 2-4 vs. 1). Regardless of the specific feature alignment strategy
(_i.e._ S-MMD, S-DANN, and our proposed cross-domain Cycle Consistency),
additional feature alignment improves performance.

We also remark that during model design, we tested variations of Cycle
Consistency-based alignment on Sim10K$\to$Cityscapes by i) conditioning on
_predicted_ rather than ground truth class and box coordinates (66.1 mAP@50,
-1.1 compared to our method), and ii) conditioning on predicted box
coordinates and ignoring class predictions (64.9 mAP@50, roughly on par with
mixing). These two settings yielded 66.1 mAP@50 (-1.1 versus Row 4) and 64.9
mAP@50 (-2.3 versus Row 4), respectively. Finally, we also tested a
dissimilarity variant of our approach (_i.e._ similar to Wang et al. (2021))
instead of consistency matching for feature alignment. This approach performs
on par with Row 4 (67.3 mAP@50 on Sim10K$\to$Cityscapes), and we consequently
opted to keep cycle consistency throughout.

Table 7:  Ablating our proposed conditional reweighting strategies on
Synscapes $\to$ Cityscapes.

[][][b]  Method w/o CB w/ CB Source 19.2 20.0 Target 34.2 40.0 Mixing 39.0
46.1 Seq. FT 39.8 44.9 [][][!htbp]  Method mAP@50
$P(\mathsf{w},\mathsf{h},\mathsf{x},\mathsf{y}|C)$ 48.5 Only
$P(\mathsf{x},\mathsf{y}|C)$ 46.7 Only $P(\mathsf{w},\mathsf{h}|C)$ 48.3 None
46.6

Table 8: Ablating $P(C)$ rewt. Table 9: Ablating $P(B|C)$ rewt. Figure 5: Per-
class performance comparison of CARE to baselines on Synscapes$\to$Cityscapes.

###  4.6 CARE: Fine-grained performance analysis

Figure 6: Visualizing $P(\mathsf{w},\mathsf{h}|C)$ reweighting on
Synscapes$\to$Cityscapes. (top) Visualizing
$v(\mathsf{w},\mathsf{h}|C=\text{car})$. (bottom) Visualizing change in mAP
after $P(\mathsf{w},\mathsf{h}|C)$ reweighting for three categories (car, bus,
bike). Figure 7:  Visualizing change in dAP (lower is better) (Bolya et al.,
2020) for errors of different types using CARE, over a mixing baseline.

$\triangleright$ $P(C)$ reweighting is highly effective (Row 5 vs. 1).
Particularly on multi-class source settings (_e.g._ Synscapes and DriveSim),
$P(C)$ reweighting considerably boosts performance. Further, Table 9 (a) shows
that class balancing naturally improves the baselines as well, due to mAP
evaluating classes equally.

$\triangleright$ $P(B|C)$ reweighting is helpful (Row 7 vs. 1). Finally, we
show that additional class-conditional box reweighting consistently improves
performance across all shifts. Table 9 (b) presents results for different
formulations of $P(B|C)$. It validates our reweighing scheme which decomposes
box size with $P(\mathsf{w},\mathsf{h}|C)$ and location with
$P(\mathsf{x},\mathsf{y}|C)$. Capturing both components is better than using
only one or neither.

Using Synscapes$\to$Cityscapes, we analyze content-specific metrics to
demonstrate CARE consistently outperforms baselines on all settings and not
just in aggregate.

$\triangleright$ CARE improves over baselines on all classes. Fig. 5 studies
per-class performance improvements with our proposed method against baselines.
Our method outperforms each baseline for every class.

$\triangleright$ CARE improves per-class performance across box sizes. Fig. 6
(_top_) visualizes bounding box frequency ratio weights
$v(\mathsf{w},\mathsf{h}|C)$ for the “car” class estimated via the first term
of Eq. (4). Matching our intuition (see Fig. 2, _right_), these ratios upweigh
target cars of sizes that are relatively less frequent in the source domain.
Fig. 6 (_bottom_) illustrates the change in mAP as a result of our reweighing
for three categories over boxes of different sizes. Here, reweighing
consistently improves mAP and can yield up to $+10$ mAP improvement for large
objects such as buses. We remark that these trends also hold for the remaining
categories.

$\triangleright$ Fine-grained error analysis. We use the TIDE (Bolya et al.,
2020) toolbox to evaluate specific error types of our mixing baseline and CARE
models (lower is better). Fig. 7 shows that CARE reduces classification,
localization, and duplicate errors, while slightly worsening joint
classification+localization errors.

$\triangleright$ Visualizing matching with cycle consistency. Fig. 4 provides
a qualitative visualization of the matching behavior of our proposed cycle
consistency approach, for two pairs of source and target images. For each
example, we estimate the Euclidean distance in feature space between all
cross-domain instance pairs in the aligned feature space of our CARE model and
visualize the closest pair of car instances for each example. As expected, we
find that our method embeds similar looking cars closer in feature space.

##  5 Discussion

We study supervised Sim2Real adaptation applied to object detection, and
propose a strategy that exploits target labels to explicitly estimate and
bridge the sim2real appearance and content gaps. Our method possesses a clear
theoretical intuition and our empirical analyses validate our improvements in
every setting that we tested, for example by boosting mAP@50 by as much as
$\sim$25%. Most importantly, this paper tackles a large research-practice gap
by bridging the literature on unsupervised and few-shot domain adaptation with
an industry-standard practice of combining labeled data from both simulated
and real domains. With this, we envision a renewed future methodological
interest in SDA.

Limitations. Our method requires sufficient labeled data in source and target
domains to reliably estimate dataset-level statistics. Further, our
formulation assumes conditional independence of box sizes and locations as
well as an equivalence between pixel-level and feature-level distributions. We
also rely on successful cross-domain alignment. These assumptions may be
violated to varying degrees in practice. We focus on object detection and the
applicability of our method to other tasks, while plausible, is not
established. Finally, we do not consider an unlabeled portion of the target
domain and leave that exploration to future work.

## References

  * Acuna et al. (2021a) Acuna, D., Philion, J., and Fidler, S.  Towards optimal strategies for training self-driving perception models in simulation.  In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , 2021a.  URL https://openreview.net/forum?id=ZfIO21FYv4.
  * Acuna et al. (2021b) Acuna, D., Zhang, G., Law, M. T., and Fidler, S.  f-domain adversarial learning: Theory and algorithms.  In Meila, M. and Zhang, T. (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 66–75. PMLR, 18–24 Jul 2021b.
  * Bolya et al. (2020) Bolya, D., Foley, S., Hays, J., and Hoffman, J.  Tide: A general toolbox for identifying object detection errors.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pp. 558–573. Springer, 2020.
  * Chattopadhyay et al. (2022) Chattopadhyay, P., Sarangmath, K., Vijaykumar, V., and Hoffman, J.  Pasta: Proportional amplitude spectrum training augmentation for syn-to-real domain generalization.  _arXiv preprint arXiv:2212.00979_ , 2022.
  * Chen et al. (2018) Chen, Y., Li, W., Sakaridis, C., Dai, D., and Van Gool, L.  Domain adaptive faster r-cnn for object detection in the wild.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3339–3348, 2018.
  * Cordts et al. (2016) Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3213–3223, 2016.
  * Csurka et al. (2022) Csurka, G., Volpi, R., and Chidlovskii, B.  Unsupervised domain adaptation for semantic image segmentation: a comprehensive survey.  _Foundations and Trends in Computer Graphics and Vision_ , 2022.
  * Donahue et al. (2013) Donahue, J., Hoffman, J., Rodner, E., Saenko, K., and Darrell, T.  Semi-supervised domain adaptation with instance constraints.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 668–675, 2013.
  * Dwibedi et al. (2019) Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zisserman, A.  Temporal cycle-consistency learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 1801–1810, 2019.
  * Ganin & Lempitsky (2015) Ganin, Y. and Lempitsky, V.  Unsupervised domain adaptation by backpropagation.  In _International Conference on Machine Learning_ , pp. 1180–1189, 2015.
  * Gao et al. (2022) Gao, Y., Yang, L., Huang, Y., Xie, S., Li, S., and Zheng, W.-S.  Acrofod: An adaptive method for cross-domain few-shot object detection.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ , pp. 673–690. Springer, 2022.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hoffman et al. (2013) Hoffman, J., Rodner, E., Donahue, J., Darrell, T., and Saenko, K.  Efficient learning of domain-invariant image representations.  _arXiv preprint arXiv:1301.3224_ , 2013.
  * Hoffman et al. (2018) Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A., and Darrell, T.  Cycada: Cycle-consistent adversarial domain adaptation.  In _International conference on machine learning_ , pp. 1989–1998. Pmlr, 2018.
  * Johnson-Roberson et al. (2017) Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S. N., Rosaen, K., and Vasudevan, R.  Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 746–753. IEEE, 2017.
  * Kar et al. (2019) Kar, A., Prakash, A., Liu, M.-Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A., and Fidler, S.  Meta-sim: Learning to generate synthetic datasets.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 4551–4560, 2019.
  * Karpathy (2021) Karpathy, A.  Tesla ai day 2021 -simulation, 2021.  URL https://www.youtube.com/watch?v=j0z4FweCy4M&t=5692s.
  * Khindkar et al. (2022) Khindkar, V., Arora, C., Balasubramanian, V. N., Subramanian, A., Saluja, R., and Jawahar, C.  To miss-attend is to misalign! residual self-attentive feature alignment for adapting object detectors.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 3632–3642, 2022.
  * Kim et al. (2019) Kim, S., Choi, J., Kim, T., and Kim, C.  Self-training and adversarial background regularization for unsupervised domain adaptive one-stage object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 6092–6101, 2019.
  * Kishore et al. (2021) Kishore, A., Choe, T. E., Kwon, J., Park, M., Hao, P., and Mittel, A.  Synthetic data generation using imitation training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3078–3086, 2021.
  * Kulis et al. (2011) Kulis, B., Saenko, K., and Darrell, T.  What you saw is not what you get: Domain adaptation using asymmetric kernel transforms.  In _CVPR 2011_ , pp. 1785–1792. IEEE, 2011.
  * Li et al. (2022) Li, Y.-J., Dai, X., Ma, C.-Y., Liu, Y.-C., Chen, K., Wu, B., He, Z., Kitani, K., and Vajda, P.  Cross-domain adaptive teacher for object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7581–7590, 2022.
  * Long et al. (2015) Long, M., Cao, Y., Wang, J., and Jordan, M.  Learning transferable features with deep adaptation networks.  In _International Conference on Machine Learning_ , pp. 97–105, 2015.
  * Long et al. (2018) Long, M., Cao, Z., Wang, J., and Jordan, M. I.  Conditional adversarial domain adaptation.  _Advances in neural information processing systems_ , 31, 2018.
  * Mahmood et al. (2022) Mahmood, R., Lucas, J., Alvarez, J. M., Fidler, S., and Law, M. T.  Optimizing data collection for machine learning.  _arXiv preprint arXiv:2210.01234_ , 2022.
  * NVIDIA (2021) NVIDIA.  Nvidia drivesim, 2021.  URL https://developer.nvidia.com/drive/simulation.
  * Prabhu et al. (2021) Prabhu, V., Khare, S., Kartik, D., and Hoffman, J.  Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8558–8567, 2021.
  * Prakash et al. (2019) Prakash, A., Boochoon, S., Brophy, M., Acuna, D., Cameracci, E., State, G., Shapira, O., and Birchfield, S.  Structured domain randomization: Bridging the reality gap by context-aware synthetic data.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pp. 7249–7255. IEEE, 2019.
  * Prakash et al. (2021) Prakash, A., Debnath, S., Lafleche, J.-F., Cameracci, E., Birchfield, S., Law, M. T., et al.  Self-supervised real-to-sim scene generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 16044–16054, 2021.
  * Ramamonjison et al. (2021) Ramamonjison, R., Banitalebi-Dehkordi, A., Kang, X., Bai, X., and Zhang, Y.  Simrod: A simple adaptation method for robust object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3570–3579, 2021.
  * Rempe et al. (2022) Rempe, D., Philion, J., Guibas, L. J., Fidler, S., and Litany, O.  Generating useful accident-prone driving scenarios via a learned traffic prior.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 17305–17315, 2022.
  * Ren et al. (2015) Ren, S., He, K., Girshick, R., and Sun, J.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Resnick et al. (2022) Resnick, C., Litany, O., Kar, A., Kreis, K., Lucas, J., Cho, K., and Fidler, S.  Causal scene bert: Improving object detection by searching for challenging groups of data.  _arXiv preprint arXiv:2202.03651_ , 2022.
  * RoyChowdhury et al. (2019) RoyChowdhury, A., Chakrabarty, P., Singh, A., Jin, S., Jiang, H., Cao, L., and Learned-Miller, E.  Automatic adaptation of object detectors to new domains using self-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 780–790, 2019.
  * Saenko et al. (2010) Saenko, K., Kulis, B., Fritz, M., and Darrell, T.  Adapting visual category models to new domains.  In _European conference on computer vision_ , pp. 213–226. Springer, 2010.
  * Saito et al. (2019a) Saito, K., Kim, D., Sclaroff, S., Darrell, T., and Saenko, K.  Semi-supervised domain adaptation via minimax entropy.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 8050–8058, 2019a.
  * Saito et al. (2019b) Saito, K., Ushiku, Y., Harada, T., and Saenko, K.  Strong-weak distribution alignment for adaptive object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6956–6965, 2019b.
  * Scott (2015) Scott, D. W.  _Multivariate density estimation: theory, practice, and visualization_.  John Wiley & Sons, 2015.
  * Tremblay et al. (2018) Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., and Birchfield, S.  Training deep networks with synthetic data: Bridging the reality gap by domain randomization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pp. 969–977, 2018.
  * Tsai et al. (2016) Tsai, Y.-H. H., Yeh, Y.-R., and Wang, Y.-C. F.  Learning cross-domain landmarks for heterogeneous domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5081–5090, 2016.
  * Tzeng et al. (2017) Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T.  Adversarial discriminative domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 7167–7176, 2017.
  * Wang et al. (2019) Wang, T., Zhang, X., Yuan, L., and Feng, J.  Few-shot adaptive faster r-cnn.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7173–7182, 2019.
  * Wang et al. (2020) Wang, X., Huang, T., Gonzalez, J., Darrell, T., and Yu, F.  Frustratingly simple few-shot object detection.  In _International Conference on Machine Learning_ , pp. 9919–9928. PMLR, 2020.
  * Wang et al. (2021) Wang, X., Huang, T. E., Liu, B., Yu, F., Wang, X., Gonzalez, J. E., and Darrell, T.  Robust object detection via instance-level temporal cycle confusion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9143–9152, 2021.
  * Wrenninge & Unger (2018) Wrenninge, M. and Unger, J.  Synscapes: A photorealistic synthetic dataset for street scene parsing.  _arXiv preprint arXiv:1810.08705_ , 2018.
  * Yao et al. (2015) Yao, T., Pan, Y., Ngo, C.-W., Li, H., and Mei, T.  Semi-supervised domain adaptation with subspace learning for visual recognition.  In _CVPR_ , 2015.
  * Zhong et al. (2022) Zhong, C., Wang, J., Feng, C., Zhang, Y., Sun, J., and Yokota, Y.  Pica: Point-wise instance and centroid alignment based few-shot domain adaptive object detection with loose annotations.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 2329–2338, 2022.
  * Zhu et al. (2019) Zhu, X., Pang, J., Yang, C., Shi, J., and Lin, D.  Adapting object detectors via selective cross-domain alignment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 687–696, 2019.

##  Appendix A Appendix

###  A.1 When is adding simulated data most helpful?

Figure 8:  Plotting the scaling behavior of mixing, seq. finetuning, and
target only baselines using 100% of source data and a varying amount of target
data, on Sim10K$\to$Cityscapes.

Intuitively, collecting real instances corresponding to the long-tail of
driving scenarios is extremely challenging, and simulated data can offer
inexpensive, unlimited labeled data to augment our dataset, albeit with an
additional domain gap. However the utility of such additional data may depend
on the nature and amount of labeled target available. A natural question is
then: in what situations can additional simulated data help boost target
performance?

To study this, we benchmark simple baselines on the Sim10K$\to$Cityscapes
shift for car detection. We plot the performance of training using only target
data, mixing, and sequential finetuning strategies as we vary the amount of
target data. For mixing and sequential finetuning we additionally use 100% of
source data. As Figure 8 shows, both baselines improve upon target only
training, with sequential finetuning initially outperforming mixing. However,
with the relatively small target task that we study, gains over target-only
training are clearly more pronounced in the low target data regime (+4.3
mAP@50 at 25% with Seq. FT), and performance saturates as we acquire more
labeled target data. For Sim10K$\to$Cityscapes, atleast with naive
combinations of simulated and real data, we find that adding simulated data
has maximum relative utility in the limited target label regime and
diminishing returns thereafter. However, even performance with target-only
training saturates towards the end, and it is unclear if the diminishing
returns are a consequence of that. Further, it is unclear whether more
principled combinations of sim and real data (via CARE) will exhibit similar
trends. Nevertheless, to rigorously study the supervised DA setting we reduce
the number of target data points (in our experiments, we randomly subsample
25%$\sim$625 examples of the target train dataset and use it for adaptation,
while leaving the test set unchanged).

$\triangleright$ Varying mixing ratio. The mixing ratio controls the
proportion of source and target data that are contained within a minibatch of
fixed size. In Fig. 9, we vary the mixing ratio of real:sim data and measure
the subsequent performance on the Cityscapes test set. We use all source and
target data for these experiments. We observed that the performance is fairly
stable beyond a ratio of 50% and, for simplicity, we adopt this mixing ratio
for all experiments unless otherwise stated.

###  A.2 Additional details on Importance reweighting

When applying bounding-box importance reweighting we introduce a smoothing
mechanism to ensure bounded loss values and thresholding to handle areas with
low target support. Specifically, we compute:

$v\left(B\mid C\right)=\begin{cases}\alpha\sigma\left(\frac{P_{T}\left(B\mid
C\right)}{P_{S}\left(B\mid C\right)}\right)+\beta&\text{ if }P_{T}\left(B\mid
C\right)>\tau\\\ 1.0&\text{ otherwise }\end{cases}$

where $\alpha,\beta$ are scaling parameters (that we set to 20, -9,
effectively bounding loss weights between 1 and 11). $\sigma$ denotes the
sigmoid operator, and $\tau$ is a probability threshold that we set to 0.1.
For boxes with very small target support, we simply set weights to a floor
value of 1.0.

###  A.3 Additional CARE analysis: Visualizing KDE estimates

In Figure 10, we visualize log PDF values from KDE estimates fit to bounding
box width and heights on both the source (Synscapes) and target (Cityscapes)
domains. As seen, the KDE is able to capture the difference in box size
distributions across domains: car class sizes vary significantly more in the
target domain, consistent with our observation in Fig. 2. It is intuitive that
with appropriate importance reweighting for source boxes as proposed in Sec.
3, we can improve overall performance across categories.

Figure 9: Sim10K$\to$Cityscapes (100% target data): Varying within-batch
real:sim ratio for mixing. Figure 10:  Visualizing log PDF values of KDE
densities fitted to bounding box size on Synscapes$\to$Cityscapes for the
“car” class.