# Multi-Source Domain Adaptation for Object Detection
with Prototype-based Mean-teacher

Atif Belal1, Akhil Meethal1, Francisco Perdigon Romero2, Marco Pedersoli1,
Eric Granger1
1 LIVIA, École de technologie supérieure, Montreal, Canada
2 GAIA Montreal, Ericsson Canada
{atif.belal.1<EMAIL_ADDRESS><EMAIL_ADDRESS>
{marco.pedersoli<EMAIL_ADDRESS>
###### Abstract

Adapting visual object detectors to operational target domains is a
challenging task, commonly achieved using unsupervised domain adaptation (UDA)
methods. When the labeled dataset is coming from multiple source domains,
treating them as separate domains and performing a multi-source domain
adaptation (MSDA) improves the accuracy and robustness over mixing these
source domains and performing a UDA, as observed by recent studies in MSDA.
Existing MSDA methods learn domain invariant and domain-specific parameters
(for each source domain) for the adaptation. However, unlike single-source UDA
methods, learning domain-specific parameters makes them grow significantly
proportional to the number of source domains used. This paper proposes a novel
MSDA method called Prototype-based Mean-Teacher (PMT), which uses class
prototypes instead of domain-specific subnets to preserve domain-specific
information. These prototypes are learned using a contrastive loss, aligning
the same categories across domains and separating different categories far
apart. Because of the use of prototypes, the parameter size of our method does
not increase significantly with the number of source domains, thus reducing
memory issues and possible overfitting. Empirical studies show PMT outperforms
state-of-the-art MSDA methods on several challenging object detection
datasets. Our code is available at https://github.com/imatif17/Prototype-Mean-
Teacher

##  1 Introduction

|
---|---

|

(a) Mean-teacher with blending of sources [15]. A standard mean-teacher model
with a single student and teacher network, trained by blending data from
multiple sources. Domain-invariant features are obtained from the backbone
trained with a discriminator. No domain-specific information is considered.

(b) Divide-and-Merge Spindle Network (DMSN) [35]. MSDA with multiple student
networks and a single-teacher network. Domain-invariant features are obtained
from the first half of the backbone (BB1). Domain-specific features are
obtained from BB 2 and the detection head for each source domain.

(c) Target-Relevant Knowledge Preservation (TRKP) [29]. MSDA with multiple
teacher networks and a single student network. Domain-invariant features are
obtained from the backbone by reversing detection head gradients. Domain-
specific features are obtained from the detection heads.

(d) Our proposed PMT method. The original mean teacher framework with one
student and one teacher is used. Domain-invariant features are obtained from
the backbone trained with a discriminator. Domain-specific information is
obtained from domain-specific prototypes for each class.

Figure 1: Comparison of the MSDA architectures using the mean-teacher method
in the case with 2 source domains. While existing methods need domain-specific
parameters (typically a detection head for each source domain) for preserving
domain information, our method keeps domain-specific information by using
prototypes for each class and domain.

Object detection (OD), one of the fundamental tasks in computer vision, has
made significant progress in recent years [40, 2, 26]. However, this progress,
typically measured by performance on curated benchmark datasets such as MS-
COCO [16] or Pascal VOC [6], may decline significantly with changes in the
data distribution between training (source) and testing (target) domain data
[39, 33]. This distribution shift can be due to many factors like variations
in weather, illumination, geographic locations, size of objects, etc. Since it
is too costly to annotate data collected from every operational target domain,
researchers have explored a multitude of techniques for unsupervised domain
adaptation (UDA), which seeks to find a common representation space between
data distributions of a labeled source domain and an unlabeled target domain
[3, 24, 15].

Among the state-of-art UDA strategies for OD, feature alignment [3, 41] and
pseudo-labeling of target data [12, 13] are the most common. Recently,
impressive results were obtained by combining both strategies, where the
popular mean-teacher [25] framework is used to pseudo-label target domain
images, and a GRL [7] integrated domain discriminator is used for alignment of
features in an adversarial way [15, 1, 5]. Mainstream UDA methods rely on a
single source domain with labeled data during adaptation. However, in a
practical setting, several source datasets may be available, and the datasets
may also be collected from multiple different domains, e.g., sensors,
environment, etc. The simplest way to use UDA with multiple sources of
training data is by blending source domain data to form a single labeled
source dataset, as shown in Fig. 1(a). However, considering each domain as a
separate source allows the OD model to explicitly address domain discrepancies
between individual source distributions, and aligning sources during
adaptation has been found to provide a higher level of recognition accuracy
and robustness [38]. This setting is referred to as multi-source domain
adaptation (MSDA).

Two methods have been proposed for MSDA in OD: Divide-and-Merge Spindle
Network (DMSN) [35] and Target-Relevant Knowledge Preservation (TRKP) [29].
The overall architecture of both methods can be divided into domain-general
and domain-specific model parameters. The initial part of the network learns
the domain-invariant representation, while the latter part learns the domain-
specific representation. This allows the model to align the features from
multiple domains while preserving domain-specific information. As illustrated
in Fig. 1(b), DMSN relies on a mean-teacher training framework, with domain-
specific student subnets for each source domain. In contrast, as shown in Fig.
1(c), TRKP proposed an adversarial disentanglement method and preserved
domain-specific knowledge using a separate detection head for each source
domain. The detection results from the teacher’s detection heads are combined
to generate pseudo-labels for the student, trained on the target domain. To
fully exploit the potential of multiple sources, both methods fundamentally
advocate for learning domain-specific weights associated with each source
domain. However, learning domain-specific weights for each source domain leads
to MSDA architectures that are difficult to train for OD since the amount of
domain-specific parameters increases rapidly with the number of sources.
Moreover, it is also observed that the weighted combination of the source
domains, relying on the heuristics of domain similarities (each pair of
source-target domains similarity quantified by a weight value) is not optimal
[32]. Additionally, the two methods focused on finding a common representation
between domains through adversarial training [35] and adversarial
disentanglement [29]. They ignored the class-wise alignment of object
categories. Objects can have variations in appearance, scale, orientation, or
other visual characteristics, which may be specific to a particular domain. If
the objects in the source and target domain exhibit significant differences,
it becomes difficult to align the features extracted by the model to
effectively detect the objects in the target domain. With the increase in the
number of domains in MSDA, this problem becomes more significant. To solve
these issues, we propose to learn domain-specific class prototype vectors. In
this paper, we argue that MSDA can be simplified without learning domain-
specific parameters for each source domain. Instead, learning domain-specific
class prototype vectors is sufficient to preserve domain-specific information
for each source domain. Additionally, because of the use prototypes, the
parameter size of our method does not increase significantly with the number
of source domains, thus reducing memory issues and possible overfitting on the
source data.

In this paper, a new MSDA method is introduced for OD that is based on mean-
teacher [18] and adversarial training (see Fig. 1(d)). The new Prototype Mean
Teacher (PMT) method relies on a cost-effective architecture that stores class
and domain specific information using prototypes. To learn domain-invariant
features, a multi-class domain discriminator is integrated into the features
generated from the OD backbone. To preserve the domain-specific information of
all domains, PMT learns class prototypes using the prototype network for all
the domains. Additionally, The prototypes are trained using contrastive loss
to perform a class-conditional and domain-specific adaptation of the detector.
The resulting architecture can provide higher OD accuracy than state-of-the-
art MSDA approaches while being conceptually simpler.

Our main contributions are summarized as follows:
(1) A novel prototype-based mean tester (PMT) method is introduced for MSDA of
object detectors. Our approach shows that prototype vectors can elegantly and
efficiently store domain-specific information.
(2) By using a multi-domain discriminator, and a contrastive loss on
prototypes for each domain and class, the proposed PMT can perform class- and
domain-conditional adaptation, improving a detector’s performance.
(3) Our experimental results and ablations show that the PMT method provides a
solution that scales well to the number of source domains, and can outperform
state-of-the-art MSDA methods for OD on benchmark datasets.

##  2 Related Works

Many DL models have been proposed for OD, namely, Faster-RCNN (FRCNN)[23],
Fully Convolutional One Stage (FCOS)[26], You Only Look Once (YOLO)[22],
Single Shot Detector (SSD)[17]. Although our PMT method is independent of the
OD model, this paper focuses on FRCNN as our base detector for SOTA
comparison.

(a) Unsupervised Domain Adaptation. UDA methods seek to alleviate the problem
of domain shift by adapting the detector trained on a labeled source dataset
to a target domain using an unlabeled target dataset. Minimizing the domain
discrepancy and adversarial learning are the most popular approaches for UDA.
In [3], the authors integrated a domain discriminator into Faster-RCNN to
learn domain-invariant feature representations. Later, [24] proposed strong
and weak alignment loss. [10] proposed a two-step domain alignment using cycle
GAN to mitigate the impact of domain shift between the source and target
domains. [34] proposed a graph-based prototype alignment, using contrastive
loss. [15] utilized a combination of adversarial training and the mean teacher
paradigm. All these methods were designed for UDA from a single source domain.
When the source data itself is coming from multiple domains, modeling the
domain discrepancy improves the accuracy and robustness of the target OD model
[29, 35]. In our work, we focus on MSDA methods for OD that explicitly
consider domain discrepancies among source datasets.

(b) Multi-Source Domain Adaptation. In the MSDA setting, the labeled training
data belongs to multiple source domains, and the MSDA method aims to distill
the knowledge from these source domains to the target domain. Only two methods
have been proposed for this setting called DMSN [35] and TRKP [29].
Fundamentally, both methods preserve domain-specific information by utilizing
specific subnets for each source domain. Thus the parameters of these methods
increase linearly as the number of source domains increases. In our method, we
preserve domain-specific information using prototypes for each domain which
results in a constant size for the MSDA model regardless of the number of
source domains.

(c) Prototype-based Learning. This learning paradigm is used in different
forms for open-world OD [11], semi-supervised OD [14], domain-adaptive OD
[37], and few-shot OD [28]. In open-world OD, prototypes are used to achieve
class separation in the feature space and help unknown class identification
[11]. In semi-supervised OD, prototypes are used for class distribution
alignment between pseudo-labels and their highly overlapping proposals [14].
In the detection of objects across domains, prototypes are used to align the
foreground and background regions between the source and target domains [37].
Few-shot OD used universal prototypes to learn the invariant object
characteristics across seen and novel classes [28]. In contrast, we use
prototypes in MSDA settings to simplify domain-specific feature learning.
Existing MSDA methods learn domain-specific features by using separate
networks for each source domain. We show that by using domain-specific
prototype vectors instead, we can avoid learning separate networks for each
domain.

##  3 Proposed Method

In the MSDA setting, we assume that there are $N$ source domains
$S_{1},S_{2},…,S_{N}$ and one target domain $T$. Each source domain can be
represented as $S_{i}=\\{(x^{j}_{i},y^{j}_{i})\\}^{M_{j}}_{i=1}$, where
$j=1,2,...,N$ indicates the source domain, and where $M_{j}$ indicates the
number of images in the source domain $j$. Here, $x^{j}_{i}$ represents the
input image $i$ of domain $j$, and $y^{j}_{i}$ represents its corresponding
annotations (bounding boxes and object categories). The target domain $T$ can
be represented as $T=\\{{\\{x_{i}\\}^{M_{T}}_{i=1}}\\}$, where $M_{T}$
indicates the number of images in the target domain. Here, $x_{i}$ represents
the input image $i$ in the target domain. In the work, we assume that all
domains have the same $K$ object categories.

The detailed architectural diagram of our PMT method is illustrated in Fig. 2.
We use the mean teacher framework with unbiased teacher [18] for semi-
supervised OD as our base model. Training of our model is performed in two
stages. First, a supervised-only training called the burn-in stage is
performed to help achieve reliable pseudo-labels on the target domain because
the teacher network is initialized using these weights. Second, the model is
adapted with both labeled data from the source domains and unlabeled data from
the target domain. With the source data, we continue using the supervised
training (as in the burn-in stage). Target domain pseudo-labels are obtained
from the teacher to train the model on the target domain. To produce domain
invariant features, a discriminator is introduced in the network.
Additionally, the prototype network is integrated to preserve domain-specific
information, and class-conditioned adaptation is performed using a contrastive
loss. The rest of this section provides details on the individual components
of our proposed PMT.

Figure 2: Architectural diagram of the proposed Prototype Mean Teacher (PMT)
for MSDA. A standard mean-teacher training framework is used. The student is
trained with backpropagation, while the teacher is an exponential moving
average of the student. The student is trained with images from all domains,
and feature alignment is performed at both the image and instance levels using
a discriminator and a prototype, respectively. During inference, the teacher
model is employed.

###  3.1 Supervised Learning

In our problem setting, the annotations are available for the source domains,
so we use them to train the model in a supervised way. For this, we compute
the standard supervised detection loss of the object detector:

$\mathcal{L}_{\mbox{sup}}=\sum_{j=1}^{N}\sum_{i=1}^{M_{j}}\mathcal{L}_{\mbox{cls}}(x^{j}_{i},y^{j}_{i})+\mathcal{L}_{\mbox{reg}}(x^{j}_{i},y^{j}_{i})$
(1)

where $\mathcal{L}_{\mbox{cls}}$ is the cross-entropy loss used for bounding
box classification, and $\mathcal{L}_{\mbox{reg}}$ is the smooth-L1 loss used
for the bounding box regression as in FRCNN.

###  3.2 Learning with Pseudo-labels

Since annotations are not available for the target domain data, we cannot
directly proceed with supervised training as with the source domain. We obtain
pseudo-labels for images for the target domain using the mean-teacher method
[25]. For that, two augmented versions of the target-domain images are
created, called weak and strong augmentations. The weak augmentation is simply
the image rescaling and horizontal flip transformation. The strong
augmentation includes color jittering, grayscale, Gaussian blur, and cutout
patches, which perform only pixel-level transformations. We followed the scale
ranges provided in [18] for the strong augmentation. Then the weak and strong
augmented versions are processed by the teacher and student networks,
respectively, as shown in Fig. 2. The predictions made by the teacher model
are used as pseudo-labels for the training of the student model. To avoid
noisy pseudo-labels from the teacher model, confidence thresholding is applied
to the teacher network’s prediction while computing the pseudo-labels. The
used unsupervised training loss for the target domain using labels from the
teacher model is:

$\displaystyle\mathcal{L}_{\mbox{unsup}}=\sum_{i=1}^{M_{T}}\mathcal{L}_{\mbox{cls}}(x_{i},\widetilde{y}_{i})+\mathcal{L}_{\mbox{reg}}(x_{i},\widetilde{y}_{i})$
(2)

where $\widetilde{y}_{i}$ represents the filtered pseudo-labels generated by
the teacher model.

###  3.3 Domain-invariant Features with Discriminator

The feature representation of a model consists of two parts: domain-invariant
and domain-specific. In a domain adaptation problem, we try to promote the
domain-invariant feature space. To achieve this, UDA methods add a binary
domain discriminator to the output of the backbone [15, 3]. This can be
extended to MSDA, by having multiple such discriminators. But this doesn’t
consider the domain discrepancy between the source domain and increases the
computational cost [38, 19]. Instead of using multiple binary discriminators,
we overcome these issues by introducing a multi-class discriminator. This
discriminator is connected to the network using a Gradient Reversal Layer
(GRL) [7]. It seeks to classify images according to domain. When this
classification loss is back-propagated, the GRL layer reverses the gradients,
so the backbone network will try to increase the classification loss,
challenging the ability of the discriminator to distinguish between domains.
Gradually, the domain of the image-level features extracted by the backbone
network will be indistinguishable from the discriminator. This simple
adversarial game between the discriminator and the backbone allows us to learn
domain-invariant features [3]. The discriminator is trained with the cross-
entropy loss between the $\hat{y}^{j}$, the $i$-th one-hot vector for the
domain $j$, and the output of the domain discriminator on image $x^{j}_{i}$:

$\displaystyle\mathcal{L}_{\mbox{dis}}=-\sum_{j=1}^{N+1}\sum_{i=1}^{M_{i}}\hat{y}^{j}\log(D(G(F(x_{i}^{j})))$
(3)

where Let $F$ is the CNN feature extractor, $G$ is the gradient reverse layer
and $D$ is the multi-domain discriminator. The summation is taken from $j=1$
to $N+1$ as we also consider the target domain.

###  3.4 Domain-specific Features with Prototypes

In our PMT method, prototypes are used to preserve domain-specific
information. To obtain the prototypes, we added a prototype network to the
output of the ROI pool features. This network maps the output of the ROI pool
to a $d$ dimensional feature space, where class-conditioned feature alignment
is performed using a contrastive loss. This aligns representations of the same
object categories across different domains and separates representations of
different object categories. Fig. 3 illustrates the working of the prototypes
for domain-specific feature alignment. We produce a local prototype for each
class (e.g., car, truck, bus in Fig. 3) in each domain, and these prototypes
preserve the domain-specific information. The prototype for each class “across
all domains”, called as global prototypes, is obtained as the mean of the
local prototypes of that class. The contrastive loss on global prototypes
pushes them apart in the feature space, resulting in better separation of the
classes. It also aligns the local prototypes from all domains for each class
reducing confusion due to appearance variation.

Figure 3: Prototype-based feature alignment with multiple source domains.
There are three domains and three classes. Each domain has a prototype for
each class. Initially, there is confusion between classes and the intra-class
distance to global prototypes from multiple domains is also large. After
alignment, class confusion and intra-class distance to global prototypes are
reduced.

Let $p_{k}^{j}$ denote the local prototype of the $k$th class in domain $j$.
The global prototype of class $k$ is denoted as $P_{k}$. Let the number of
occurrences of class $k$ from domain $j$ in a minibatch be $c_{k}^{j}$. The
prototype update value for class $k$ from domain $j$ in a minibatch is
computed as

$q_{k}^{j}=\frac{1}{|c_{k}^{j}|}\sum_{r=1}^{c_{k}^{j}}{P}({R}({F}(x^{j}),r))$
(4)

where $P$ is the prototype network (a 2 layers MLP), $R$ is the fasterRCNN
ROI-pooling and $F$ is the CNN feature extractor. Thus, ${R}({F}(x^{j}),r)$
pools features from the RoI $r$ corresponding to ground-truth box for class
$k$ from domain $j$ in a minibatch. For the target domain without ground-truth
annotations, the pseudo-labels obtained from the teacher network are used to
compute the update value $q_{k}^{N+1}$. When $c_{k}^{j}=0$ in a minibatch, no
updates are performed for the corresponding prototype. The local prototype
vector $p_{k}^{j}$ for each class $k$ and domain $j$ is stored in memory along
with a count value $\rho_{k}^{j}$ that tells how many times $p_{k}^{j}$ has
been updated thus far. Given the current update value $q_{k}^{j}$, local
prototype $p_{k}^{j}$ is updated as:

$p_{k}^{j}=\frac{\rho_{k}^{j}p_{k}^{j}+q_{k}^{j}}{\rho_{k}^{j}+1}$ (5)

The count of updates $\rho_{k}^{j}$ is incremented after this operation. After
updating the prototype vectors, the contrastive loss is computed in two parts.
The first part aligns the prototypes of the same class from different domains
by maximizing their similarity. The second part pushes apart the prototypes of
different classes to reduce class confusion. To align the prototypes of the
same class from different domains, we need to consider pairwise combinations
of the domains: $C=Comb(N+1,2)$ (including source and target domains). Let
$\mathcal{C}$ be the cosine similarity between two vectors. The alignment of
category $k$ across different domains is achieved by maximizing the following
similarity:

$\frac{1}{CK}\sum_{k=1}^{K}\sum_{j=1}^{N+1}\sum_{l\neq
j}^{N+1}\mathcal{C}(p_{k}^{j},p_{k}^{l})$ (6)

normalized by the total number of pairwise similarities $CK$. To push the
prototypes of the different classes apart, we minimize their cosine
similarity. For this, we first compute the global prototypes from local
prototypes of each domain as their weighted mean:

$P_{k}=\frac{\sum_{j=1}^{N}\rho_{k}^{j}p_{k}^{j}}{\sum_{j=1}^{N}\rho_{k}^{j}}$
(7)

where the weight $\frac{\rho_{k}^{j}}{\sum_{j=1}^{N}\rho_{k}^{j}}$ assigns
importance to the prototype of class $k$ from domain $j$ according to its
number of occurrence $\rho_{k}^{j}$. Once $P_{k}$ for each class is computed,
we maximize the following similarity function:

$\sum_{k=1}^{K}\sum_{l\neq k}^{K}\mathcal{C}(P_{k},P_{l})$ (8)

Note that prototypes from the target domain are not considered here because of
the noise they induce due to the use of pseudo-labels. The final contrastive
loss on prototypes is:

$\displaystyle\mathcal{L}_{\mbox{prot}}=\sum_{k=1}^{K}\sum_{l\neq
k}^{K}\mathcal{C}(P_{k},P_{l})-$ (9)
$\displaystyle\frac{1}{CK}\sum_{k=1}^{K}\sum_{j=1}^{N+1}\sum_{l\neq
j}^{N+1}\mathcal{C}(p_{k}^{j},p_{k}^{l})$

###  3.5 Training Summary

The FRCNN model is initialized with Imagenet pre-trained weights. The weights
of the student model are updated using back-propagation, while the weights of
the teacher model are obtained as the exponential moving average of the
student model’s weights over time. During training, first, we start our burn-
in training stage. In this step only the Eq. 1 is used to train the student
model. After the burn-in stage, the teacher model is initialized as an exact
copy of the student model. From now on, the weights of the student model are
adapted using the back-propagation, while the weights of the teacher are the
EMA of the student model’s weights. In the unsupervised MSDA training step,
all the losses discussed above are used jointly to adapt the student model.
Therefore, the total loss is:

$\displaystyle\mathcal{L}=\mathcal{L}_{\mbox{sup}}+\alpha\mathcal{L}_{\mbox{unsup}}+\beta\mathcal{L}_{\mbox{dis}}+\gamma\mathcal{L}_{\mbox{prot}}$
(10)

where $\alpha$, $\beta$, and $\gamma$ are hyperparameters to weight each loss.

##  4 Results and Discussion

###  4.1 Experimental Methodology

Implementation Details. Our experiments follow the same procedure as in [29,
35]. FRCNN[23] VGG16 backbone pre-trained with ImageNet was used as the
detection framework. Similar to [9], ROI-alignment was used, and the shorter
side of the input image was reduced to 600 pixels. For our mean-teacher
training, the same setting was used as in [18]. For filtering the pseudo-
labels a threshold of 0.7 is used. The model is trained in the burn-in setting
for 15 epochs. Then, the model was trained in the unsupervised DA setting for
another 15 epochs. For all the settings, the value $\alpha$ and $\beta$ were
set to 1 and 0.1, respectively, while the value of $\gamma$ was set to 1.2,
0.6, and 0.1 for Sections 4.2, 4.3 and 4.4. The weight smoothing coefficient
of EMA is set to 0.9996. We conducted all the experiments on 4 A100 GPUs, with
batch size 4 and a learning rate of 0.2. Our approach was implemented using
Pytorch[20] and Detectron2[30].

Quantitative comparison. We compared our approach with the following baseline:
(1) Source-Only: FRCNN[23] is trained in a supervised manner on source domain
data, and no adaptation is performed. (2) UDA Blending: all the source domain
data is combined, and UDA methods [24, 34, 31, 18, 15] are used to adapt
FRCNN. (3) MSDA: adaptation of FRCNN from independent source datasets. It
includes MSDA methods developed for classification [21, 38] and upgraded for
OD, as well as MSDA methods developed for OD [35][29]. (4) Oracle: Target-Only
and All-Combined. In the Target-Only case, we performed supervised training of
FRCNN only on labeled target data. In the All-Combined case, supervised
training is performed on all source and target data combined. Note that some
of the baseline results are taken from [35] and [29].

###  4.2 Cross Time Adaptation

In this setting, images are collected at different times of the day, so domain
shift is caused by changes in illumination. The performance of our method in
this setting is evaluated on the BDD100K[36] dataset. The dataset consists of
data collected at three different times - Daytime, Dusk/Dawn, and Night Time.
In our experiments, Daytime and Night are used as the source domains, and
Dusk/Dawn as the target domain. The source domain consists of labeled 64,699
(36,728 daytime + 27,971 night) images. In the target domain, we have 5,027
unlabeled images, that are used for training. The evaluation is done on 778
validation set images of Dusk/Dawn. The mAP on 10 classes is reported.

The mAP performance of our method is compared against the other approaches in
Table 1. Class-wise AP is reported in Table 7 in Appendix. It can be observed
that the UDA methods that blend source datasets are able to increase the
performance of the detector compared to the Source-Only baseline. However,
this increase in performance is not large, as they disregard the inter-source
domain shift. It is also interesting to note that two state-of-the-art MSDA
classification methods [38, 21], perform worse than the Source-Only baseline.
This degradation in performance shows the fundamental difference in MSDA
methods for classification and detection and a strong motivation for MSDA for
OD. Our method improves on the performance of the best-performing MSDA method
[29] by 5.5%. It can also be seen that the performance of our PMT method is
much better than the Oracle Target-Only case (because of the small size of the
target dataset), and it approached the mAP of the Oracle All-Combined case.

Setting | Method | D | N | D+N
---|---|---|---|---
Source Only | FRCNN[23] | 30.4 | 25.0 | 28.9
UDA Blending | Strong-Weak[24] | 31.4 | 26.9 | 29.9
Graph Prototype[1] | 31.8 | 27.6 | 30.6
Cat. Regularization[31] | 31.2 | 28.4 | 30.2
UMT[5] | 33.8 | 21.6 | 33.5
Adaptive Teacher[15] | 34.9 | 27.8 | 34.6
MSDA | MDAN[38] | - | - | 27.6
M3SDA[21] | - | - | 26.5
DMSN[35] | - | - | 35.0
TRKP[29] | - | - | 39.8
| PMT(ours) | - | - | 45.3
Oracle | Target-Only | - | - | 26.6
All-Combined | - | - | 45.6

Table 1: Detection AP of PMT compared against the baseline, UDA, MSDA, and
oracle methods on BDD100K. Source domains are daytime (D) and night (N)
subsets and the target is always Dusk/Dawn of BDD100K.

###  4.3 Cross Camera Adaptation

In this setting, the data are collected using different cameras, and domain
shift is caused by changes in the camera’s resolution and viewpoint. In this
setting BDD100K[36], Cityscape[4], and Kitty[8] datasets are used. For our
experiments, we used Cityscape and Kitty as the source domains, while the
Daytime domain of BDD100k was used as the target domain. For both training and
evaluation, we only considered the images with car objects. This provides a
source domain that consists of 9,515 (2,831 Cityscapes + 6,684 Kitty) labeled
images. In the target domain, we have 36,728 unlabeled images used for
training. The evaluation is done on 5,258 validation set images of Daytime.
The AP is reported only on the car object category in Table 2.

Note that in this experiment, there is only one object category (car), so we
cannot use both components of our contrastive loss 9. For this case, the
prototype separation part was removed for our contrastive loss. It can be
observed that the variation in the performance is similar to Section 4.2. Our
PMT can outperform state-of-the-art methods, but the increase in AP is lower
compared to TRKP. We suppose this is due to the removal of the prototype
separation part of our contrastive loss. In ablation studies, we showed that
every loss component contributed to obtaining the best PMT performance.

Setting | Method | C | K | C+K
---|---|---|---|---
Source Only | FRCNN[23] | 44.6 | 28.6 | 43.2
UDA Blending | Strong-Weak[24] | 45.5 | 29.6 | 41.9
Cat. Regularization[31] | 46.5 | 30.8 | 43.6
UMT[5] | 47.5 | 35.4 | 47.0
Adaptive Teacher[15] | 49.8 | 40.1 | 48.4
MSDA | MDAN[38] | - | - | 43.2
M3SDA[21] | - | - | 44.1
DMSN[35] | - | - | 49.2
TRKP[29] | - | - | 58.4
| PMT(ours) | - | - | 58.7
Oracle | Target-Only | - | - | 60.2
All-Combined | - | - | 69.7

Table 2: AP for the car class. Our proposed method PMT is compared against the
baseline, UDA, MSDA, and oracle methods on the Daytime domain of BDD100K
dataset. C and K refer to Cityscapes and Kitty datasets.

###  4.4 Extension to Mixed Domain Adaptation

In the MSDA, the domain shift among source data is not always limited to one
factor. So, a setting with domain shift with multiple factors is considered
for validation. In the source domain, we considered the MS COCO[16],
Cityscapes[4], and Synscapes[27] datasets, and the Daytime domain of the
BDD100K dataset are the target domain. Among the source domains, the domain
shift is mixed and there are more classes, making this study a challenging
scenario. Also, the number of source domains is increased from two to three.
The source domain has 99,724 (2,975 Cityscapes + 71,749 MS COCO + 25,000
Synscapes) labeled images and, the target domain has 36,728 unlabeled images.
The evaluation is performed on 5,258 validation set images of Daytime. For
training and evaluation, we only employ the images having 7 common object
categories between the datasets. From the results reported in Table 3, we can
observe that the Source-Only setting performs better than the UDA Blending
setting. This is because of the complex domain shift among the source data.
When only Cityscapes and MS COCO are used as the source domain, our method
outperforms the previous best-performing method by 3.4%. After adding the
Synscapes dataset our method still outperforms other methods, including the
Oracle Target-Only results. This experiment further highlights the
effectiveness of our method. Class-wise AP is reported in Table 8 in Appendix.

Setting | Method | C | C + M | C + M + S
---|---|---|---|---
Source Only | FRCNN[23] | 23.4 | 29.7 | 30.9
UDA Blending | Unbiased Teach.[18] | - | 18.5 | 25.1
Adaptive Teach.[15] | - | 22.9 | 29.6
MSDA | TRKP[29] | - | 35.3 | 37.1
PMT(ours) | - | 38.7 | 39.7
Oracle | Target-Only | - | - | 38.6
All-Combined | - | 47.1 | 48.2

Table 3:  mAP on 7 object categories of our PMT compared against baselines on
the Daytime domain of BDD100K. C, M, and S refer to Cityscapes, MS COCO, and
Synscapes datasets.

###  4.5 Ablation Studies

Impact of the different components. The cross-time adaptation setting is used
in the ablation studies. The two main components of our method are the
prototype network and multi-class discriminator. To clearly disentangle the
effect of terms in the contrastive loss for prototypes, we considered them as
two separate parts: “prototype separation” and “prototype alignment” part. The
prototype separation loss increases the distance between different object
categories across domains, whereas the prototype alignment loss aligns the
same object category across various domains. The experimental results shown in
Table 4 show that the results without using the discriminator and prototype
are better than the Source-Only result of Table 1. This is due to the
effectiveness of the mean-teacher framework in MSDA. By adding the
discriminator to the network, performance is increased by 10.2%. This result
is already better than the previous state of art MSDA methods for OD. By
adding both of our prototype loss terms individually, the performance of the
model further increases. Note that using the alignment loss performs better
than the separation loss because the separation loss only discriminates the
categories, and the vanilla FRCNN is already doing it. Finally, when both the
prototype loss terms are combined, the performance is further improved.

Multidomain | Prototype | Prototype |
---|---|---|---
Discriminator | Separation | Alignment | $\textrm{AP}_{50}$
|  |  | 31.7
$\checkmark$ |  |  | 41.9
$\checkmark$ | $\checkmark$ |  | 43.0
$\checkmark$ |  | $\checkmark$ | 43.4
$\checkmark$ | $\checkmark$ | $\checkmark$ | 44.6

Table 4: Ablation study on the components of our PMT method.

###  4.6 Parameter Growth with Number of Domains

One of the important advantages of not using domain-specific weights is the
reduction in the number of parameters to learn. While the existing MSDA
detectors have a significant linear growth in parameters with the number of
domains, the parameter growth of our method is negligible as the number of
source domains increases. Tab. 5 illustrates this. DMSN [35] makes some part
of the feature extractor also domain-specific, thus showing a significantly
higher rate of growth compared to other methods. TRKP [29] makes only the
detection heads domain-specific, so the growth rate is less compared to DMSN.
Our method has only domain-specific prototypes which require fewer parameters
per domain than the other approaches.

Method | Number of source domains
---|---
1 | 2 | 3 | 4 | 5
DMSN [35] | 45.994 | 75.426 | 104.858 | 134.290 | 163.722
TRKP [29] | 45.994 | 59.942 | 73.890 | 87.838 | 101.786
PMT (ours) | 46.586 | 46.587 | 46.588 | 46.589 | 46.590

Table 5: Model parameter growth (in millions) as the number of source domains
increases. While the parameters of DMSN [35] and TRKP [29] grow quickly with
the number of source domains, the parameters of our method PMT remain almost
constant because the increase is only due to the prototype vectors.

##  5 Conclusion

A commonly used architecture for MSDA is to learn domain-invariant and domain-
specific parameters (for each source domain) to effectively adapt DL models
from multiple source domains. In this paper, we proposed a method for
simplifying MSDA where domain-specific information is retained using class
prototypes for each domain. This avoids the need to train a domain-specific
subnet for each source domain, simplifying the MSDA architecture
significantly. The parameters of our model remain almost constant regardless
of the number of source domains used as only prototype vectors are needed for
each domain. Thus, our model size is comparable to UDA methods which perform
adaptation from a single source. Experimental results show that our proposed
method can effectively exploit multiple domains and improve on the state-of-
the-art for multi-source object detection.

## References

  * [1] Qi Cai, Yingwei Pan, Chong-Wah Ngo, Xinmei Tian, Lingyu Duan, and Ting Yao.  Exploring object relation in mean teacher for cross-domain detection, 2019.
  * [2] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [3] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Domain adaptive faster r-cnn for object detection in the wild, 2018.
  * [4] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [5] Jinhong Deng, Wen Li, Yuhua Chen, and Lixin Duan.  Unbiased mean teacher for cross-domain object detection, 2021.
  * [6] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 2010.
  * [7] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation, 2015.
  * [8] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [9] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn, 2018.
  * [10] Han-Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung-Yu Tseng, Maneesh Singh, and Ming-Hsuan Yang.  Progressive domain adaptation for object detection, 2019.
  * [11] K J Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian.  Towards open world object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), 2021.
  * [12] Mehran Khodabandeh, Arash Vahdat, Mani Ranjbar, and William G. Macready.  A robust learning approach to domain adaptive object detection, 2019.
  * [13] Seunghyeon Kim, Jaehoon Choi, Taekyung Kim, and Changick Kim.  Self-training and adversarial background regularization for unsupervised domain adaptive one-stage object detection.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6091–6100, 2019.
  * [14] Aoxue Li, Peng Yuan, and Zhenguo Li.  Semi-supervised object detection via multi-instance alignment with global class prototypes.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [15] Yu-Jhe Li, Xiaoliang Dai, Chih-Yao Ma, Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He, Kris Kitani, and Peter Vajda.  Cross-domain adaptive teacher for object detection.  In CVPR, 2022.
  * [16] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick.  Microsoft COCO: Common objects in context.  In ECCV, 2014.
  * [17] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: Single shot MultiBox detector.  In Computer Vision – ECCV 2016, pages 21–37. Springer International Publishing, 2016.
  * [18] Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda.  Unbiased teacher for semi-supervised object detection, 2021.
  * [19] G. Park and S. Wan Lee.  Information-theoretic regularization for multi-source domain adaptation.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9194–9203, 2021.
  * [20] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [21] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation, 2019.
  * [22] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection, 2016.
  * [23] S. Ren, K. He, R. Girshick, and J. Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [24] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko.  Strong-weak distribution alignment for adaptive object detection, 2019.
  * [25] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In NeurIPS, 2018.
  * [26] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection, 2019.
  * [27] Magnus Wrenninge and Jonas Unger.  Synscapes: A photorealistic synthetic dataset for street scene parsing, 2018.
  * [28] Aming Wu, Yahong Han, Linchao Zhu, Yi Yang, and Cheng Deng.  Universal-prototype augmentation for few-shot object detection.  2021\.
  * [29] Jiaxi Wu, Jiaxin Chen, Mengzhe He, Yiru Wang, Bo Li, Bingqi Ma, Weihao Gan, Wei Wu, Yali Wang, and Di Huang.  Target-relevant knowledge preservation for multi-source domain adaptive object detection, 2022.
  * [30] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [31] Chang-Dong Xu, Xing-Ran Zhao, Xin Jin, and Xiu-Shen Wei.  Exploring categorical regularization for domain adaptive object detection, 2020.
  * [32] Minghao Xu, Hang Wang, and Bingbing Ni.  Graphical modeling for multi-source domain adaptation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  * [33] Minghao Xu, Hang Wang, Bingbing Ni, Qi Tian, and Wenjun Zhang.  Cross-domain detection via graph-induced prototype alignment.  In CVPR, pages 12355–12364, 2020.
  * [34] Minghao Xu, Hang Wang, Bingbing Ni, Qi Tian, and Wenjun Zhang.  Cross-domain detection via graph-induced prototype alignment, 2020.
  * [35] Xingxu Yao, Sicheng Zhao, Pengfei Xu, and Jufeng Yang.  Multi-source domain adaptation for object detection, 2021.
  * [36] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning, 2020.
  * [37] Yixin Zhang, Zilei Wang, and Yushi Mao.  Rpn prototype alignment for domain adaptive object detector.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [38] Han Zhao, Shanghang Zhang, Guanhang Wu, José M. F. Moura, Joao P Costeira, and Geoffrey J Gordon.  Adversarial multiple source domain adaptation.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [39] Y. Zheng, D. Huang, S. Liu, and Y. Wang.  Cross-domain object detection through coarse-to-fine feature adaptation.  In CVPR, 2020.
  * [40] X. Zhou, R. Girdhar, A. Joulin, and P. Krähenbühl andd I. Misra.  Detecting twenty-thousand classes using image-level supervision.  In ECCV, 2022.
  * [41] Xinge Zhu, Jiangmiao Pang, Ceyuan Yang, Jianping Shi, and Dahua Lin.  Adapting object detectors via selective cross-domain alignment.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 687–696, 2019.

##  6 Appendix

###  6.1 Impact of $\gamma$ on different settings

The main component responsible for simplifying the MSDA problem with our
approach is prototype-based learning. In this study, we investigate the
importance of contrastive loss on prototypes by studying the impact of
$\gamma$ in different settings. For experiments, we considered the two cases;
Cross Time Adaptation and Mixed Domain Adaptation settings. The Cross Time
Adaptation setting is much simpler compared to the Mixed Domain Adaptation
setting. By choosing these settings, we get an idea of the effect of $\gamma$
on simple and complex MSDA problems. The analysis result is reported in Table
6. It can be observed that for the Cross Time Adaptation setting, the
performance of our method increased with the increase in $\gamma$, while it
decreased for the Mixed Domain Adaptation setting. With the increase in value
of $\gamma$, the model is forced to learn a class-conditioned aligned feature
space. Learning this feature space is much easier if the domain shift among
the datasets is not much. But with the increase in domain shift, learning the
aligned feature space becomes difficult.

$\gamma$ | Source Domain | Target Domain | mAP
---|---|---|---
0.1 | D+N | Dusk/Dawn | 43.6
0.5 | 43.9
0.9 | 44.6
1.2 | 45.3
1.5 | 45.1
0.1 | C+M+S | Daytime | 39.7
0.5 | 39.1
0.9 | 38.7
1.2 | 38.3
1.5 | 38.2

Table 6: Effect of $\gamma$ under different domain shift conditions.

###  6.2 Impact of the Number of Classes

In this ablation, we consider the effect of a different number of object
categories for training and evaluation on our PMT. We analyze the performance
of a given class when trained with a varying number of other classes and how
this affects the final detection performance. As the prototypes are trained
with a contrastive loss, we expect that an increased number of classes could
help to further improve the overall adaptation. In Fig. 4, we present the AP
for the classes Person and Car when trained with 0 to 8 additional classes and
Truck and Raider when trained with 2 to 8 extra classes. While for some of the
classes (Person and Car), the effect is not clear, for others (Truck and
Rider) there is a clear trend in which more classes are used and better the
results. We hypothesize that this different behavior could be due to the
number of training samples per class. For very common classes with many
samples, the model is already quite general, and adding more classes does not
help. Instead for classes with fewer data points, contrasting the prototypes
with other classes helps to improve detection.

Figure 4: AP of PMT with a growing number of object categories.

Setting | Source | Method | Bike | Bus | Car | Motor | Person | Rider | Light | Sign | Train | Truck | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source Only | D | FRCNN[23] | 35.1 | 51.7 | 52.6 | 9.9 | 31.9 | 17.8 | 21.6 | 36.3 | - | 47.1 | 30.4
N | 27.9 | 32.5 | 49.4 | 15.0 | 28.7 | 21.8 | 14.0 | 30.5 | - | 30.7 | 25.0
D+N | 31.5 | 46.9 | 52.9 | 8.4 | 29.5 | 21.6 | 21.7 | 34.3 | - | 42.2 | 28.9
UDA Blending | D+N | Strong-Weak[24] | 29.7 | 50.0 | 52.9 | 11.0 | 31.4 | 21.1 | 23.3 | 35.1 | - | 44.9 | 29.9
Graph Prototype[1] | 31.7 | 48.8 | 53.9 | 20.8 | 32.0 | 21.6 | 20.5 | 33.7 | - | 43.1 | 30.6
Cat. Regularization[31] | 25.3 | 51.3 | 52.1 | 17.0 | 33.4 | 18.9 | 20.7 | 34.8 | - | 47.9 | 30.2
UMT[5] | 42.3 | 48.1 | 56.4 | 13.5 | 35.3 | 26.9 | 31.1 | 41.7 | - | 40.1 | 33.5
Adaptive Teacher[15] | 43.1 | 48.9 | 56.9 | 14.7 | 36.0 | 27.1 | 32.7 | 43.8 | - | 42.7 | 34.6
MSDA | D+N | MDAN[38] | 37.1 | 29.9 | 52.8 | 15.8 | 35.1 | 21.6 | 24.7 | 38.8 | - | 20.1 | 27.6
M3SDA[21] | 36.9 | 25.9 | 51.9 | 15.1 | 35.7 | 20.5 | 24.7 | 38.1 | - | 15.9 | 26.5
DMSN[35] | 36.5 | 54.3 | 55.5 | 20.4 | 36.9 | 27.7 | 26.4 | 41.6 | - | 50.8 | 35.0
TRKP[29] | 48.4 | 56.3 | 61.4 | 22.5 | 41.5 | 27.0 | 41.1 | 47.9 | - | 51.9 | 39.8
PMT(ours) | 55.3 | 59.8 | 67.6 | 29.9 | 47.6 | 32.7 | 46.3 | 56.0 | - | 57.7 | 45.3
Oracle | D+N | Target Only | 27.2 | 39.6 | 51.9 | 12.7 | 29.0 | 15.2 | 20.0 | 33.1 | - | 37.5 | 26.6
All-Combined | 56.4 | 59.9 | 67.3 | 30.8 | 47.9 | 33.9 | 47.2 | 57.8 | - | 54.8 | 45.3

Table 7: Class-wise AP of PMT compared against the baseline, UDA, MSDA, and
oracle methods on BDD100K. Source domains are daytime (D) and night (N)
subsets and the target is always Dusk/Dawn of BDD100K

Setting | Source | Method | Person | Car | Rider | Truck | Motor | Bicycle | Bike | mAP
---|---|---|---|---|---|---|---|---|---|---
Source Only | C | FRCNN[23] | 26.9 | 44.7 | 22.1 | 17.4 | 17.1 | 18.8 | 16.7 | 23.4
Source Only | C+M | FRCNN[23] | 35.2 | 49.5 | 26.1 | 25.8 | 18.9 | 26.1 | 26.5 | 29.7
UDA Blending | Unbiased Teach.[18] | 30.7 | 28.0 | 3.9 | 11.2 | 19.2 | 17.8 | 18.7 | 18.5
UDA Blending | Adaptive Teacher[15] | 31.2 | 31.7 | 15.1 | 16.4 | 17.1 | 20.9 | 27.9 | 22.9
MSDA | TRKP[29] | 39.2 | 53.2 | 32.4 | 28.7 | 25.5 | 31.1 | 37.4 | 35.3
MSDA | PMT(ours) | 41.1 | 53.5 | 31.2 | 31.9 | 33.7 | 34.9 | 44.6 | 38.7
Source Only | C+M+S | FRCNN[23] | 36.6 | 49.0 | 22.8 | 24.9 | 26.9 | 28.4 | 27.7 | 30.9
UDA Blending | Unbiased Teach.[18] | 32.7 | 39.6 | 6.6 | 21.2 | 21.3 | 25.7 | 28.5 | 25.1
UDA Blending | Adaptive Teacher[15] | 36.3 | 42.6 | 19.7 | 23.4 | 24.8 | 27.1 | 33.2 | 29.6
MSDA | TRKP[29] | 40.2 | 53.9 | 31.0 | 30.8 | 30.4 | 34.0 | 39.3 | 37.1
MSDA | PMT(ours) | 43.3 | 54.1 | 32.0 | 32.6 | 35.1 | 36.1 | 44.8 | 39.7
Oracle | C+M | Target Only | 35.3 | 53.9 | 33.2 | 46.3 | 25.6 | 29.3 | 46.7 | 38.6
C+M | All-Combined | 40.2 | 60.1 | 47.1 | 60.0 | 29.2 | 36.3 | 56.9 | 47.1
C+M+S | All-Combined | 41.7 | 63.9 | 49.5 | 58.1 | 31.6 | 39.1 | 53.5 | 48.2

Table 8: Class-wise AP of PMT compared against the baselines on Daytime domain
of BDD100K. C, M, and S refer to Cityscapes, MS COCO, and Synscapes datasets.


f^{\prime}\rVert_{\infty}$.

$\lVert\cdot\rVert_{n}$: The empirical $L^{2}$–norm $\lVert
f\rVert_{n}^{2}=\sum_{k=1}^{n}f(X_{k\Delta})^{2}$.

## References

  * [1] Yannick Baraud.  A Bernstein-type inequality for suprema of random processes with applications to model selection in non-Gaussian regression.  Bernoulli, 16(4):1064–1085, 2010.
  * [2] Richard F Bass.  Stochastic Processes.  Cambridge University Press, New York, 1st edition, 2011.
  * [3] Rabi Bhattacharya, Manfred Denker, and Alok Goswami.  Speed of convergence to equilibrium and to normality for diffusions with multiple periodic scales.  Stochastic Processes and their Applications, 80(1):55–86, 1999\.
  * [4] Lucien Birgé and Pascal Massart.  Minimum contrast estimators on sieves: exponential bounds and rates of convergence.  Bernoulli, 4(3):329–375, 1998.
  * [5] Ismaël Castillo and Richard Nickl.  Nonparametric Bernstein–von Mises theorems in Gaussian white noise.  The Annals of Statistics, 41(4):1999–2028, 2013.
  * [6] Ismaël Castillo and Richard Nickl.  On the Bernstein–von Mises phenomenon for nonparametric Bayes procedures.  The Annals of Statistics, 42(5):1941–1969, 2014.
  * [7] Jakub Chorowski.  Statistics for diffusion processes with low and high-frequency observations.  PhD thesis, Humboldt-Universität zu Berlin, Mathematisch-Naturwissenschaftliche Fakultät, 2016.
  * [8] Fabienne Comte, Valentine Genon-Catalot, and Yves Rozenholc.  Penalized nonparametric mean square estimation of the coefficients of diffusion processes.  Bernoulli, 13(2):514–543, 2007.
  * [9] Arnak Dalalyan.  Sharp adaptive estimation of the drift function for ergodic diffusions.  The Annals of Statistics, 33(6):2507–2528, 2005.
  * [10] Subhashis Ghosal, Jayanta Ghosh, and Aad van der Vaart.  Convergence rates of posterior distributions.  The Annals of Statistics, 28(2):500–531, 2000.
  * [11] Subhashis Ghosal and Aad van der Vaart.  Convergence rates of posterior distributions for noniid observations.  The Annals of Statistics, 35(1):192–223, 2007.
  * [12] Subhashis Ghosal and Aad van der Vaart.  Fundamentals of nonparametric Bayesian inference.  Cambridge University Press, Cambridge, 2017.
  * [13] I Gihman and A Skorohod.  Stochastic Differential Equations.  Springer Verlag, Berlin, 1972.
  * [14] Evarist Giné and Richard Nickl.  Rates of contraction for posterior distributions.  The Annals of Statistics, 39(6):2883–2911, 2011.
  * [15] Evarist Giné and Richard Nickl.  Mathematical Foundations of Infinite-Dimensional Statistical Models.  Cambridge University Press, Cambridge, 2016.
  * [16] Emmanuel Gobet, Marc Hoffmann, and Markus Reiß.  Nonparametric estimation of scalar diffusions based on low frequency data.  The Annals of Statistics, 32(5):2223–2253, 2004.
  * [17] Shota Gugushvili and Peter Spreij.  Nonparametric Bayesian drift estimation for multidimensional stochastic differential equations.  Lithuanian Mathematical Journal, 54(2):127–141, 2014.
  * [18] Marc Hoffmann.  Adaptive estimation in diffusion processes.  Stochastic Processes and their Applications, 79(1):135–163, 1999\.
  * [19] Mathieu Kessler.  Estimation of an Ergodic Diffusion from Discrete Observations.  Scandinavian Journal of Statistics, 24(2):211–229, 1997.
  * [20] Yury A. Kutoyants.  Statistical Inference for Ergodic Diffusion Processes.  Springer London, London, 2004.
  * [21] Robert S. Liptser and Albert N. Shiryaev.  Statistics of Random Processes : I. General Theory.  Springer Berlin Heidelberg, 1st edition, 1977.
  * [22] Richard Nickl.  Bernstein–von Mises theorems for statistical inverse problems I: Schrödinger equation.  Journal of the European Mathematical Society, to appear.
  * [23] Richard Nickl and Jakob Söhl.  Nonparametric Bayesian posterior contraction rates for discretely observed scalar diffusions.  The Annals of Statistics, 45(4):1664–1693, 2017.
  * [24] Omiros Papaspiliopoulos, Yvo Pokern, Gareth O. Roberts, and Andrew M. Stuart.  Nonparametric estimation of diffusions: a differential equations approach.  Biometrika, 99(3):511–531, 2012.
  * [25] Daniel Paulin.  Concentration inequalities for Markov chains by Marton couplings and spectral methods.  Electronic Journal of Probability, 20, 2015.
  * [26] Y Pokern, A M Stuart, and J H van Zanten.  Posterior consistency via precision operators for Bayesian nonparametric drift estimation in SDEs.  Stochastic Processes and their Applications, 123(2):603–628, 2013\.
  * [27] David Pollard.  A User’s Guide to Measure Theoretic Probability.  Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2001.
  * [28] Kolyan Ray.  Bayesian inverse problems with non-conjugate priors.  Electronic Journal of Statistics, 7(1):2516–2549, 2013.
  * [29] L.C.G. Rogers and David Williams.  Diffusions, Markov processes and martingales. Vol.2, Ito calculus.  Cambridge mathematical library. Cambridge University Press, Cambridge, 2nd edition, 2000.
  * [30] Jakob Söhl and Mathias Trabs.  Adaptive confidence bands for Markov chains and diffusions: Estimating the invariant measure and the drift.  ESAIM: Probability and Statistics, 20:432–462, 2016.
  * [31] Hans Triebel.  Theory of function spaces.  Monographs in mathematics. Birkhäuser Verlag, 1983.
  * [32] Frank van der Meulen, Moritz Schauer, and Jan van Waaij.  Adaptive nonparametric drift estimation for diffusion processes using Faber–Schauder expansions.  Statistical Inference for Stochastic Processes, 2017.
  * [33] Frank van der Meulen and Harry van Zanten.  Consistent nonparametric Bayesian inference for discretely observed scalar diffusions.  Bernoulli, 19(1):44–63, 2013.
  * [34] Jan van Waaij and Harry van Zanten.  Gaussian process methods for one-dimensional diffusions: Optimal rates and adaptation.  Electronic Journal of Statistics, 10(1):628–645, 2016.