#  A Comparative Study of Federated Learning Models for COVID-19 Detection

Erfan Darzidehkalani1, Nanna M. Sijtsema1, P.M.A van Ooijen1
1Machine learning lab, Data Science Center in Health (DASH)
University of Groningen, Hanzeplein 1, Groningen, The Netherlands
<EMAIL_ADDRESS>This paper is being prepared with IEEE standards This
work was funded in part by NWO under project AMICUSErfan Darzidehkalani is
with UMCG (e-mail: e.darzidehkalani@umcg.nl).

###### Abstract

Deep learning is effective in diagnosing COVID-19 and requires a large amount
of data to be effectively trained. Due to data and privacy regulations,
hospitals generally have no access to data from other hospitals. Federated
learning (FL) has been used to solve this problem, where it utilizes a
distributed setting to train models in hospitals in a privacy-preserving
manner. Deploying FL is not always feasible as it requires high computation
and network communication resources. This paper evaluates five FL algorithms’
performance and resource efficiency for Covid-19 detection. A decentralized
setting with CNN networks is set up, and the performance of FL algorithms is
compared with a centralized environment. We examined the algorithms with
varying numbers of participants, federated rounds, and selection algorithms.

Our results show that cyclic weight transfer can have better overall
performance, and results are better with fewer participating hospitals. Our
results demonstrate good performance for detecting COVID-19 patients and might
be useful in deploying FL algorithms for covid-19 detection and medical image
analysis in general.

###### Index Terms:

federated learning, medical image analysis, COVID-19, privacy preserving
machine learning

##  I Introduction

Coronaviruses are a family of viruses that cause respiratory and intestinal
illnesses in humans and animals. The best-known variants are those responsible
for the COVID-19, SARS, and MERS epidemics. Some people with COVID-19 will
develop serious complications including COVID pneumonia, which can be
recognized in lung CT scans. Research has shown the effectiveness of chest
imaging in diagnosing COVID-19-infected people. Deep learning methods, such as
Convolutional neural networks (CNN), can help radiologists diagnose COVID-19
with severe symptoms in various image analysis tasks[1]. For Covid-19 deep
learning, models have shown great promise in spotting infected areas in CT
scans and X-ray images.

The training of deep learning models requires sufficient and diverse medical
datasets gathered from multiple data holders. And most of the existing
solutions rely on a central entity in charge of collecting data from different
hospitals. However, medical images may contain confidential and sensitive
information about patients that often cannot be shared outside the
institutions of their origin. One potential solution to this problem is
federated deep learning. FL aims to decentralize the whole process of training
by keeping the data locally. In FL, the algorithm training is performed in a
decentralized manner by different nodes, or clients, that use local data. In
this scenario, each decentralized node trains an individual model using its
data and shares the model parameters (instead of the data) with the rest.

FL can differ from centralized data sharing in a number of ways. While both
approaches aim to optimize their learning objective, FL algorithms have to
account for the fact that communication with clients takes place over
unreliable networks with very limited upload speeds. So unlike the centralized
setting in which computation is generally a bottleneck, in FL communication
might be the bottleneck.

In this paper, we developed a framework that enables collaboration between
hospitals and uses multiple data sources to detect COVID-19 infection using FL
The decentralized way of distributing data among different centers guarantees
privacy and data is kept locally[2].

##  II Background and Related works

Federated learning has been used for various imaging modalities such as MRI[3]
[4], X-ray [5] retinal imaging, [5] and for tasks such as brain tumor
segmentation [6] [7] diagnosis [8] and treatment selection [7]. FL has shown
great promise in developing models to support doctors in making treatment
decisions for COVID-19 patients; it was investigated and reported that FL had
a clear impact on patient care in a large-scale study on COVID-19 patients
across 20 centers on five continents[9]. They used chest X-Ray imaging data in
addition to clinical data to determine hospital triage for level of care and
oxygen requirement in COVID-19 patients. They demonstrated that FL improved
model performance for clients with limited datasets, compared to when they
were trained on their local data. Another finding was that medical centers
with smaller datasets had some classes with only a few patients resulting in
underrepresented categories. These clients saw a significant improvement in
prediction for those patient categories, which is especially important
because, in some clients, less than 5% of the COVID-19 patients were
categorized as having severe symptoms, while more than 95% had moderate
symptoms. However, their care is more critical and requires more attention.

Figure 1: Schematic view of FL models and algorithms, (a) Federated averaging,
clients train on local batch of data (b) FedSGD, a subsample of clients are
selected, and each performs single step SGD and sends the model updates to the
server (c) Cyclic Weight transfer (CWT), clients train locally, and pass the
model to the next clients, and the cycle repeats (d) Single weight transfer
(SWT) model passes each client only once. (e) Stochastic weight transfer
(STWT), the model is passed sequentially through clients, and participating
clients in each round are sampled randomly.

Several recent studies have been done to classify scan images of COVID-19
infeced patients, and healthy subjects, and to locate the lesion areas. The
primary focus of AI tooling in the management of COVID-19 patients is
interpreting radiology images, mainly chest CT, which has been widely applied
for detecting lung changes to optimize patient management, and guide treatment
decisions[10][11][12]. Other studies have investigated 3D classification
networks, [13] or Covid-19 detection with limited training samples. Most of
the above studies achieved good accuracy and assumed a centralized environment
where one data center has access to all the data.Three publications exist that
successfully applied distributed learning for COVID-19 detection[14][15][16].

The global aggregation models used in the above studies were limited to model
averaging in federated[16], [14], or blockchain setting [15]. Other studies
showed limitations of the existing algorithms like a large communication
overhead [17], and problems with convergence or catastrophic forgetting after
increasing the number of participating hospitals [3] [18]. To our knowledge,
no study has been performed that compared multiple FL algorithms under
standard conditions to evaluate their applicability. Therefore, comparing
multiple FL algorithms under standard conditions could be informative in
evaluating their applicability in practice. To evaluate the existing methods
from multiple perspectives, we have implemented the most popular models and
compared them in terms of performance, communication overhead, and computation
burden.

##  III Algorithms

Centralized data sharing In Centralized data sharing (CDS), data is stored in
a central location and can be accessed by all clients. This is in contrast to
federated and decentralized data sharing methods, where data is stored in
multiple locations and accessed by a single user or limited numbers of users.
We use CDS as a baseline for comparison with other algorithms.

Federated averaging: The learning procedure for federated averaging is an
iterative process containing local and global steps. Each data owner trains a
model received from a global server on its local dataset in local iterations
[19]. The global server updates the global model by aggregating the updated
local models. Then it sends it back to clients for the next round. The
optimization problem for federated averaging can be formulated as

$w^{t+1}=\sum\limits_{i=1}^{N}{p_{i}w_{i}^{t}},w_{i}^{t}=\arg\min\limits_{w_{i}}{\left(\mathcal{L}(\mathcal{D}_{i};w^{t})\right)}$
(1)

where $N$ is the number of data owners, $\mathcal{L}(\mathcal{D}_{i};w^{t})$
is a loss function indicating global model parameters $w^{t}$ of local
datasets, and $p_{i}$ is the probability of selecting client $i$. Local
optimization can be formulated as $w_{i}^{t+1}\leftarrow
w^{t}-\eta\cdot\nabla\mathcal{L}(w^{t};\mathcal{D}_{i})$, where $\eta$ is the
learning rate. The global model can be updated based on the local models
$w_{i}$ and is shared for aggregation:

$w^{t+1}=\sum\limits_{i=1}^{N}{p_{i}w_{i}^{t+1}}$ (2)

Federated stochastic gradient descent:Federated Stochastic Gradient Descent
(FedSGD) is a variation of Federated Averaging (FedAvg) that uses a large-
batch synchronous approach to multi-client learning. FedSGD utilizes a subset
of clients from the total number of clients, where $C$ defines the subset of
selected clients. This subset of clients is selected at each global round, and
the global server sends the most recent global model to them. Each client then
performs local training over its dataset for a select number of epochs. The
global model is updated based on the local models received from each client
and is shared for aggregation, similar to FedAvg. However, in FedSGD, the
gradient is computed over the selected batch of clients and therefore, $C<1$,
for $C=1$ the training would be non-stochastic (full batch) since all the
clients are involved. This allows for training with large batches, as the
gradient is computed over the selected subset of clients. The optimization
problem for FedSGD can be formulated as

$w^{t+1}=w^{t}-\eta\cdot\sum\limits_{i=1}^{C}p_{i}\cdot\nabla\mathcal{L}(w^{t};\mathcal{D}_{i})$
(3)

where $\eta$ is the learning rate, $p_{i}$ is the probability of selecting
client $i$ and $\mathcal{L}$ is the loss function. The key difference between
FedAvg and FedSGD lies in the use of large-batch synchronous approach in
FedSGD. This approach has been shown to outperform the naive asynchronous SGD
training due to the increased accuracy and efficiency, as compared to the
local training approach used in FedAvg [20][21]. Additionally, FedSGD has been
shown to be more robust to non-IID data distributions, compared to FedAvg
[20].

Cyclic weight transfer: Federated learning techniques have been widely used in
medical image processing tasks using a method known as cyclic weight transfer
(CWT)[5]. This method involves training models on individual clients for a
number of iterations and then cyclically sharing the updated weights with the
following client. However, the existing CWT algorithm faces a notable
challenge, as it lacks the ability to effectively manage inter-client
variability in training data or labels. To ensure the practical application of
CWT, it is crucial to develop a version that can handle the common variations
observed in a majority of real-world medical imaging datasets.[2]

Single weight transfer: Single weight transfer (SWT) is another FL model
widely used in the medical imaging domain. In Single weight transfer, models
are trained in each client with its local data, and then the updated model is
transferred to the next client. The difference between this method and CWT is
that here the model passes each client only once.

Stochastic weight transfer: In stochastic weight transfer (STWT), we select a
subsample of clients and train them in a cyclic manner. Similar to FedSGD, a
ratio defines the number of selected clients to the total number of clients in
each federated round.

##  IV Experiment

Dataset Our experiments used two publicly available data sources, the Tongji
hospital dataset and Brazil’s SARS-CoV-2 dataset. Tongji dataset consists of
349 chest CT-scans of COVID-19 positive and 397 scans of healthy subjects, all
low-resolution CT modalities. Brazil’s SARS-CoV-2 dataset consists of 2482
samples, 1252 scans of COVID-19-infected patients, and 1230 healthy subjects
collected from multiple hospitals in Sao Paulo, Brazil. Train and test sets
were obtained randomly from the aggregated datasets. Table I, shows data
distribution.

TABLE I:  Data distribution Data | Class | Dataset | Train | Test
---|---|---|---|---
Covid | Brazil | 1252 | 1451 | 150
Tongji | 349
Non-Covid | Brazil | 1230 | 1477 | 150
Tongji | 397

Preprocessing Images were selected as 2D slices in greyscale. Preprocessing
included randomly cropping between 0.5 to full size, random horizontal
flipping, and intensity normalization. CT-slices were all resized to
224$\times$224 pixels with interpolation. Figure 2 shows samples of processed
images.

Figure 2: Sample CT slices of Covid-19 images (top row) and Non-covid images
(bottom row)

Training ResNet-18 is used as the backbone deep learning model. ResNet-18
comprises one initial block cascaded to four middle blocks. The initial block
is made of convolutional, batch normalization, ReLU, and pooling layers.
Middle blocks have the same layers, connected with straight and skip
connections. The model is pre-trained on ImageNet dataset [22] with a
CrossEntropy loss function and learning rate of $0.05$. Each federated round
consisted of 20 internal epochs for each client and batches of 16 samples in
each iteration. For models which use minibatch training, like STWT and FedSGD,
a subset of clients is randomly selected. Similar to training, test data was
split into mini-batches, and the results were averaged across batches. We
performed training with various participating clients and federated rounds to
evaluate their effect on final performance. Models were also trained in a
centralized, non-federated setting to build a comparison baseline.

Figure 3: Data distribution of each client in the simulated federated setting

Evaluation Standard classification metrics, accuracy, recall, precision, and
F1 score, were used as our evaluation criteria. We also evaluated the level of
communication, the amount of transferred data in each algorithm, and the
computational complexity of the models.

##  V Results

Here, the result for the setting with 10 participating clients and a maximum
of 10 rounds is presented. The results are average performance among clients
for all the federated rounds. Table II shows the results.

TABLE II:  Comparison of FL algorithms on classification of COVID-19 data for 10 clients, averaged performance in all the 10 rounds. Method | Accuracy | Recall | Precision | F1 score
---|---|---|---|---
CDS | 87.75% | 89.57% | 87.93% | 87.19%
FedAVG | 66.72% | 70.02% | 43.80% | 51.7%
FedSGD | 65.17% | 68.24% | 43.86% | 47.75%
CWT | 87.75% | 89.00% | 88.67% | 87.52%
SWT | 64.60% | 74.33% | 65.55% | 59.66%
STWT | 84.21% | 84.09% | 83.33% | 81.71%

Effect number of federated rounds

To evaluate the effect of number of rounds, models with 3, 5, 10 and 15 rounds
were tested. The test results are shown for both centralized and FL
algorithms. Table III shows the results of our experiment. The increasing
number of rounds correlates with higher accuracy of the global model.

TABLE III:  Effect number of rounds on accuracy of FL algorithms for 10 clients, 20 internal epochs. Method | 3 rounds | 5 rounds | 10 rounds | 15 rounds
---|---|---|---|---
CDS | 85.06% | 81.56% | 91.06% | 91.04%
FedAVG | 56.05% | 63.78% | 69.64% | 70.73%
FedSGD | 50.88% | 55.9% | 75.59% | 76.94%
CWT | 80.77% | 89.78% | 91.27% | 93.56%
STWT | 90.73% | 83.97% | 89.44% | 93.01%

Effect number of participating clients To evaluate the number of clients on
the FL network, we examined scenarios with 3,5, and 8 participating clients.
We trained each of the clients in 20 internal epochs. The number of Federated
rounds for all the algorithms (except SWT) was 10. The average test results
are shown in the Figure 4.

Figure 4: Accuracy of FL algorithms with differrent number of clients.  Figure 5: Test accuracy as a function of passing rounds TABLE IV:  Computation time (seconds) for FL algorithms for standardized setting Method | 3 clients | 5 clients | 8 clients | 10 clients
---|---|---|---|---
FedAVG | 8934 sec | 8975 sec | 9002 sec | 9030 sec
FedSGD | 8810 sec | 8853 sec | 9013 sec | 9052 sec
CWT | 5119 sec | 5450 sec | 5383 sec | 5556 sec
STWT | 2805 sec | 5243 sec | 6101 sec | 6129 sec
SWT | 543 sec | 547 sec | 589 sec | 618 sec

Communication can also be a bottleneck in this setting. In methods like
federated averaging, the lower bounds for total communicated data are
proportional to $\sim$ ${2NT}$ where T is the number of rounds and N is the
number of participating clients. In CWT, this lower bound is $\sim$ ${NT}$. In
our setting, we use a ResNet 101 model. We calculated the overall transferred
data for the different number of rounds. As expected, in settings where
clients are chosen stochastically, the communication time is lower than full-
client participation. Also, the experiments for computational costs show that
non-sequential models generally have higher computational requirements than
sequential models.

TABLE V:  Comparison of total transferred data in a normalized setting in (GB) Method | 3 rounds | 5 rounds | 10 rounds | 15 rounds
---|---|---|---|---
FedAVG | 1.371 | 2.286 | 4.571 | 6.857
FedSGD | 0.823 | 1.371 | 2.743 | 4.114
CWT | 0.686 | 1.143 | 2.286 | 3.428
STWT | 0.411 | 0.686 | 1.371 | 2.057

##  VI Discussion

Our results show that FL has comparable performance to centralized data
sharing, with the advantage of keeping data private. With large volumes of
data and after high number of rounds, centralized data sharing and cyclic
weight transfer have the highest accuracy.

Sequential models are susceptible to catastrophic forgetting, where a global
model performs well on the latest client it has seen while having poor
performance in other clients. Conversely, in algorithms like FedAvg and
FedSGD, the models are averaged asynchronously after all the clients have
finished their training. So the trajectory is smoother and overall improving
with more communication rounds. As shown in Figure 5, local test results can
have a high variance when passing through clients sequentially, indicating the
catastrophic forgetting effect.

Models like FedAvg, and FedSGD, in which all the clients have identical copies
of one global model, are slower and more challenging to converge compared to
sequential models like CWT and STWT. Also, FedAvG and FedSGD require more
training resources due to active server participation, resulting in more
computation and network consumption. Stochastic client selection is an
efficient way of training. Stochastic models save significant time and
resources while having similar performance to full client participation.
Overall, CWT and STWT have best results in terms of model accuracy and
computation times. These findings could be practical in further federated
deployments in medical institutions.

Sequential models like CWT and STWT perform better than non-sequential models
on fewer training rounds. For example, after three rounds of training, STWT
and CWT both reach 96% accuracy, while FedAvG reaches 66%, and FedSGD performs
equally to a random classifier. As the training proceeds, FedAVG and FedSGD
gradually improve with more global rounds.The concept of sequential models is
similar to fine-tuning [23] in centralized deep learning, so in cases where a
hospital temporarily joins an FL network, or there is an urgency in training,
sequential models are a better option.

More training rounds do not always lead to a better global model. Although
average performance on all clients improves, more global rounds lead to worse
performance for some clients. The global model can overfit some clients,
leading to lower performance on others[24]. Some studies suggested early
stopping and fine-tuning to local dataset after global training is finished
[25]. In all the algorithms, more clients resulted in slower convergence. This
effect is stronger in the FedAvg algorithm. In FedAVG, the Global model must
compromise between potentially disparate local minima.[26] Methods such as
adaptive or stochastic selection of clients and momentum-based models help
faster convergence. [27] Our results suggest that stochastic client
participation is close to full client participation. The average results of
four trials with varying rounds, shown in Table III indicate that stochastic
client participation in FedSGD results in 5.23% performance loss and 40% less
bandwidth consumption compared to FedAvg. In STWT, it results in only 1.25%
less accuracy but saves 40% of communication and 11.3% of computation. These
results are in accordance with prior studies, showing that, in theory,
stochastic and full client participation have similar global minima[28].
Stochastic client selection can be advantageous when there are limited
resources, or in larger networks with occasionally unavailable clients.

We did not assume any shift in clients’ data. A more comprehensive analysis
should consider the effect of the domain and distribution shifts on the
performance of the algorithms. Also, inter-client data variability and the
effect of heterogenous clients could be a future line of research.

##  VII Conclusion

FL enables extensive collaborations of hospitals to address medical imaging
problems while keeping data private. Real-world implementation requires
consideration of efficiency and hardware requirements in addition to model
performance, especially in the healthcare field, which generally has limited
infrastructure. We implemented five FL algorithms for COVID-19 detection and
analyzed their efficiency and accuracy. Our results suggest that FL algorithms
have comparable performance to centralized data sharing, with the advantage of
keeping data private. They also show that the sequential methods are a better
option in most of the scenarios. This study can be helpful in the deployment
of FL systems in COVID-19 detection and medical image analysis in general.

## Acknowledgement

This research is supported by KWF Kankerbestrijding and the Netherlands
Organisation for Scientific Research (NWO) Domain AES, as part of their joint
strategic research programme: Technology for Oncology IL. The collaboration
project is co-funded by the PPP allowance made available by Health Holland,
Top Sector Life Sciences & Health, to stimulate public-private partnerships.
The authors acknowledge Nikos Sourlos for his comments.

## References

  * [1] S. Kogilavani, J. Prabhu, R. Sandhiya, M. S. Kumar, U. Subramaniam, A. Karthick, M. Muhibbullah, and S. B. S. Imam, “Covid-19 detection based on lung ct scan using deep learning techniques,” _Computational and Mathematical Methods in Medicine_ , vol. 2022, 2022.
  * [2] E. Darzidehkalani, M. Ghasemi-Rad, and P. van Ooijen, “Federated learning in medical imaging: Part ii: Methods, challenges, and considerations,” _Journal of the American College of Radiology: JACR_ , pp. S1546–1440.
  * [3] M. J. Sheller, B. Edwards, G. A. Reina, J. Martin, S. Pati, A. Kotrotsou, M. Milchenko, W. Xu, D. Marcus, R. R. Colen, _et al._ , “Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data,” _Scientific reports_ , vol. 10, no. 1, pp. 1–12, 2020\.
  * [4] S. Silva, B. A. Gutman, E. Romero, P. M. Thompson, A. Altmann, and M. Lorenzi, “Federated learning in distributed medical databases: Meta-analysis of large-scale subcortical brain data,” in _2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019)_. IEEE, 2019, pp. 270–274.
  * [5] N. Balachandar, K. Chang, J. Kalpathy-Cramer, and D. L. Rubin, “Accounting for data variability in multi-institutional distributed deep learning for medical imaging,” _Journal of the American Medical Informatics Association_ , vol. 27, no. 5, pp. 700–708, 2020.
  * [6] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” _Scientific data_ , vol. 4, no. 1, pp. 1–13, 2017.
  * [7] J. Lee, J. Sun, F. Wang, S. Wang, C.-H. Jun, and X. Jiang, “Privacy-preserving patient similarity learning in a federated environment: development and analysis,” _JMIR medical informatics_ , vol. 6, no. 2, p. e7744, 2018.
  * [8] I. Pan, H. H. Thodberg, S. S. Halabi, J. Kalpathy-Cramer, and D. B. Larson, “Improving automated pediatric bone age estimation using ensembles of models from the 2017 rsna machine learning challenge,” _Radiology: Artificial Intelligence_ , vol. 1, no. 6, p. e190053, 2019.
  * [9] M. Flores, I. Dayan, H. Roth, A. Zhong, A. Harouni, A. Gentili, A. Abidin, A. Liu, A. Costa, B. Wood, _et al._ , “Federated learning used for predicting outcomes in sars-cov-2 patients,” _Research Square_ , 2021.
  * [10] L. Yan, H.-T. Zhang, J. Goncalves, Y. Xiao, M. Wang, Y. Guo, C. Sun, X. Tang, L. Jing, M. Zhang, _et al._ , “An interpretable mortality prediction model for covid-19 patients,” _Nature machine intelligence_ , vol. 2, no. 5, pp. 283–288, 2020.
  * [11] Y. Hu, J. Jacob, G. J. Parker, D. J. Hawkes, J. R. Hurst, and D. Stoyanov, “The challenges of deploying artificial intelligence models in a rapidly evolving pandemic,” _Nature Machine Intelligence_ , vol. 2, no. 6, pp. 298–300, 2020.
  * [12] E. Burian, F. Jungmann, G. A. Kaissis, F. K. Lohöfer, C. D. Spinner, T. Lahmer, M. Treiber, M. Dommasch, G. Schneider, F. Geisler, _et al._ , “Intensive care risk estimation in covid-19 pneumonia based on clinical and imaging parameters: experiences from the munich cohort,” _Journal of clinical medicine_ , vol. 9, no. 5, p. 1514, 2020.
  * [13] X. Wang, X. Deng, Q. Fu, Q. Zhou, J. Feng, H. Ma, W. Liu, and C. Zheng, “A weakly-supervised framework for covid-19 classification and lesion localization from chest ct,” _IEEE transactions on medical imaging_ , vol. 39, no. 8, pp. 2615–2625, 2020.
  * [14] W. Zhang, T. Zhou, Q. Lu, X. Wang, C. Zhu, H. Sun, Z. Wang, S. K. Lo, and F.-Y. Wang, “Dynamic-fusion-based federated learning for covid-19 detection,” _IEEE Internet of Things Journal_ , vol. 8, no. 21, pp. 15 884–15 891, 2021\.
  * [15] R. Kumar, A. A. Khan, J. Kumar, N. A. Golilarz, S. Zhang, Y. Ting, C. Zheng, W. Wang, _et al._ , “Blockchain-federated-learning and deep learning models for covid-19 detection using ct imaging,” _IEEE Sensors Journal_ , vol. 21, no. 14, pp. 16 301–16 314, 2021.
  * [16] T.-T. Ho, K.-D. Tran, and Y. Huang, “Fedsgdcovid: Federated sgd covid-19 detection under local differential privacy using chest x-ray images and symptom information,” _Sensors_ , vol. 22, no. 10, p. 3728, 2022.
  * [17] S. W. Remedios, J. A. Butman, B. A. Landman, and D. L. Pham, “Federated gradient averaging for multi-site training with momentum-based optimizers,” in _Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning_. Springer, 2020, pp. 170–180.
  * [18] K. Chang, N. Balachandar, C. Lam, D. Yi, J. Brown, A. Beers, B. Rosen, D. L. Rubin, and J. Kalpathy-Cramer, “Distributed deep learning networks among institutions for medical imaging,” _Journal of the American Medical Informatics Association_ , vol. 25, no. 8, pp. 945–954, 2018.
  * [19] E. Darzidehkalani, M. Ghasemi-Rad, and P. van Ooijen, “Federated learning in medical imaging: Part i: Toward multicentral health care ecosystems,” _Journal of the American College of Radiology: JACR_ , pp. S1546–1440.
  * [20] D. Chai, L. Wang, K. Chen, and Q. Yang, “Fedeval: A benchmark system with a comprehensive evaluation model for federated learning,” _arXiv preprint arXiv:2011.09655_ , 2020.
  * [21] Z. Charles, Z. Garrett, Z. Huo, S. Shmulyian, and V. Smith, “On large-cohort training for federated learning,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [23] Y.-T. Chen, Y.-C. Chuang, and A.-Y. Wu, “Online extreme learning machine design for the application of federated learning,” in _2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)_. IEEE, 2020, pp. 188–192.
  * [24] M. Mohri, G. Sivek, and A. T. Suresh, “Agnostic federated learning,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 4615–4625.
  * [25] T. Yu, E. Bagdasaryan, and V. Shmatikov, “Salvaging federated learning by local adaptation,” _arXiv preprint arXiv:2002.04758_ , 2020.
  * [26] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” _arXiv preprint arXiv:1907.02189_ , 2019.
  * [27] W. Liu, L. Chen, Y. Chen, and W. Zhang, “Accelerating federated learning via momentum gradient descent,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 31, no. 8, pp. 1754–1766, 2020.
  * [28] Y. J. Cho, J. Wang, and G. Joshi, “Client selection in federated learning: Convergence analysis and power-of-choice selection strategies,” _arXiv preprint arXiv:2010.01243_ , 2020.


[table]capposition=top

11institutetext: KAIST, South Korea. 11email: {feipan, sshuh1215,
<EMAIL_ADDRESS>KENTECH, South Korea. 22email:
<EMAIL_ADDRESS>33institutetext: Harvard University, USA. 33email:
<EMAIL_ADDRESS>
# ML-BPM: Multi-teacher Learning with Bidirectional Photometric Mixing for
Open Compound Domain Adaptation in Semantic Segmentation

Fei Pan 1Paper ID 6380 1{feipan, sshuh1215<EMAIL_ADDRESS>KAIST,
South Korea.  Sungsu Hur 1Paper ID 6380 1{feipan, sshuh1215,
<EMAIL_ADDRESS>KAIST, South Korea.  Seokju Lee 2KENTECH, South Korea.
<EMAIL_ADDRESS>Junsik Kim 3Harvard University, USA<EMAIL_ADDRESS>In So Kweon 1Paper ID 6380 1{feipan, sshuh1215<EMAIL_ADDRESS>1Paper
ID 6380 1{feipan, sshuh1215<EMAIL_ADDRESS>1Paper ID 6380 1{feipan,
sshuh1215<EMAIL_ADDRESS>2KENTECH, South Korea<EMAIL_ADDRESS>3Harvard University, USA<EMAIL_ADDRESS>1Paper ID 6380 1{feipan,
sshuh1215<EMAIL_ADDRESS>
###### Abstract

Open compound domain adaptation (OCDA) considers the target domain as the
compound of multiple unknown homogeneous subdomains. The goal of OCDA is to
minimize the domain gap between the labeled source domain and the unlabeled
compound target domain, which benefits the model generalization to the unseen
domains. Current OCDA for semantic segmentation methods adopt manual domain
separation and employ a single model to simultaneously adapt to all the target
subdomains. However, adapting to a target subdomain might hinder the model
from adapting to other dissimilar target subdomains, which leads to limited
performance. In this work, we introduce a multi-teacher framework with
bidirectional photometric mixing to separately adapt to every target
subdomain. First, we present an automatic domain separation to find the
optimal number of subdomains. On this basis, we propose a multi-teacher
framework in which each teacher model uses bidirectional photometric mixing to
adapt to one target subdomain. Furthermore, we conduct an adaptive
distillation to learn a student model and apply consistency regularization to
improve the student generalization. Experimental results on benchmark datasets
show the efficacy of the proposed approach for both the compound domain and
the open domains against existing state-of-the-art approaches.

###### Keywords:

Domain Adaptation, Open Compound Domain Adaptation, Semantic Segmentation,
Multi-teacher Distillation

##  1 Introduction

Semantic segmentation is a fundamental task in finding applications to many
problems, including robotics [34], autonomous driving [35], and medical
diagnosis [16]. Recently, deep learning-based semantic segmentation approaches
[12, 36, 35] have achieved remarkable progress. However, their effectiveness
and generalization ability require a large amount of pixel-wised annotated
data which are expensive to collect. To reduce the cost of data collection and
annotation, numerous synthetic datasets have been proposed [21, 22]. However,
the models trained on synthetic data tend to poorly generalize to real images.
To cope with this issue, unsupervised domain adaptation (UDA) methods [26, 28,
37, 17, 25, 33] have proposed to align the domain gap between the source and
the target domain. Despite the efficacy of UDA techniques, most of these works
rely on the strong assumption that the target data is composed of a single
homogeneous domain. This assumption is often violated in real-world scenarios.
As an illustration in autonomous driving, the target data will likely be
composed of various subdomains such as night, snow, rain, etc. Therefore,
directly applying the current UDA approaches to these target data might
deliver limited performance. This paper focuses on the challenging problem of
open compound domain adaptation (OCDA) in semantic segmentation where the
target domain is unlabeled and contains multiple homogeneous subdomains. The
goal of OCDA is to adapt a model to a compound target domain and to further
enhance the model generalization to the unseen domains.
To perform OCDA, Liu  _et al._ [13] propose an easy-to-hard curriculum
learning strategy, where samples closer to the source domain will be chosen
first for adaptation. However, it does not fully take advantage of the
subdomain boundaries information in the compound target domain. To explicitly
consider this information, current OCDA works [8, 19] propose to separate the
target compound domain into multiple subdomains based on image style
information. Existing works use a manual domain separation method; they also
employ a single model to simultaneously adapt to all the target subdomain.
However, adapting to a target subdomain might hinder the model from adapting
to other dissimilar target subdomains, which leads to limited performance. We
propose a multi-teacher framework with bidirectional photometric mixing for
open compound domain adaptation in semantic segmentation to tackle this issue.
First, we propose automatic domain separation to find the optimal number of
subdomains and split the target compound domain. Then, we present a multi-
teacher framework in which each teacher model uses bidirectional photometric
mixing to adapt to one target subdomain. On this basis, we conduct adaptive
distillation to learn a student model and apply a fast and short online
updating using consistency regularization to improve the student’s
generalization to the open domains. We evaluate our approach on the benchmark
datasets. The proposed approach outperforms all the existing state-of-the-art
OCDA techniques and the latest UDA techniques for domain adaptation and domain
generalization task.
The Contribution of This Work. (1) we propose automatic domain separation to
find the optimal number of target subdomains; (2) we present a multi-teacher
framework with bidirectional photometric mixing to reduce the domain gaps
between the source domain and every target subdomain separately; (3) we
further conduct an adaptive distillation to learn a student model and apply
consistency regularization to improve the student generalization to the open
domains.

##  2 Related Work

Unsupervised Domain Adaptation. Unsupervised domain adaptation (UDA)
techniques are used to reduce the expensive cost of pixel-wise labeling tasks
like semantic segmentation. In UDA, adversarial learning is used actively to
align input-level style using image translation, feature distribution, or
structured output [27, 10, 28, 17, 29]. Alternatively, self-training
approaches [2, 33, 25, 37] have also recently demonstrated compelling
performance in this context. While these works have shown significant
improvement, adopting those works directly for practical usage shows
limitations due to its restricted setting dealing with only single source and
single target. Despite the improvement provided by UDA techniques, their
applicability to real scenarios remains restricted by the implicit assumption
that the target data contains images from a single distribution.

Domain Generalization. The purpose of domain generalization (DG) is to train a
model – solely using source domain data – such that it can perform reliable
predictions on unseen domain. While DG is an essential problem, a few works
have attempted to address this problem in the task of semantic segmentation.
DG for semantic segmentation shows two main streams: augmentation-based and
network-based approaches. The augmentation-based approaches [30, 11] propose
to significantly augment the training data via an additional style dataset to
learn domain-invariant representation. The network-based approaches [18, 4]
attempt to modify the structure of the network to minimize domain-specific
information (such as colors or styles) such that the resulting model mainly
focuses on the content-specific information. Even though DG for semantic
segmentation has achieve obvious progress, their performance is inevitably
lower than several UDA methods due to the absence of the target images, which
is capable of providing abundant domain-specific information.

Open Compound Domain Adaptation. Liu  _et al._ [13] firstly suggests Open
Compound Domain Adaptation (OCDA) that handles unlabeled compound
heterogeneous target domain and unseen open domain. While Liu  _et al._ [13]
propose a curriculum learning strategy, it fails to consider the specific
information of each target subdomain. Current OCDA works [8, 19] propose to
separate the compound target domain into multiple subdomains to handle the
intra-domain gaps. Gong  _et al._ [8] adopt domain-specific batch
normalization for adaptation. Park  _et al._ [19] utilize GAN-based image
translation and adversarial training to exploit domain invariant features from
multiple subdomains.

##  3 Generating Optimal Subdomains

Figure 1: The part of generating optimal subdomains consists of automatic
domain separation (ADS) and subdomain style purification (SSP). In ADS, we
adopt Silhouette Coefficient [23] to find the optimal number of subdomains
$k^{*}$. In SSP, we calculate mean of histogram $\widetilde{H}^{c}_{m}$ for
the $m\textsuperscript{th}$ target subdomain $T_{m}$ according to Equation 4,
and the purified subdomain is denoted as $\widetilde{T}_{m}$.

###  3.1 Automatic Domain Separation

Our work assumes that the domain-specific property of images comes from their
styles. Existing works adopt a predefined parameter to decide the number of
subdomains, which might lead to a nonoptimal domain adaptation performance;
furthermore, they rely on a pre-trained CNN-based encoder to extract the style
information for the subdomain discovery. However, we propose an automatic
domain separation (ADS) to effectively separate the target domain using the
distribution of pixel values of the target images. The proposed ADS is capable
of predicting the optimal number of subdomains without relying on any
predefined parameters and extracting the image style information without
relying on any pre-trained CNN models. We denote the source domain as
$\mathcal{S}$, and the unlabeled compound target domain as $\mathcal{T}$. We
also assume compound target domain contains $k$ latent subdomains:
$\\{T_{1},\dots,T_{k}\\}$, which lack of clear prior knowledge to distinguish
themselves. The goal of ADS is to find the optimal number of subdomains
$k^{*}$ and separate $\mathcal{T}$ into several subdomains accordingly.

Current work [14] suggests a simple yet effective style translation method by
matching the distribution of pixel values on LAB color space. Thus, we adopt
LAB space into ADS to extract the style information of the target image. Given
a target RGB image $x_{t}\in\mathcal{T}$ as input, we convert it into LAB
color space $rgb2lab(x_{t})$. The three channels in LAB color space are
represented as $l,a$, and $b$. Then, we compute the histograms of the pixel
values for all three channels in LAB color space: $H^{l}(x_{t}),H^{a}(x_{t})$,
and $H^{b}(x_{t})$. The histograms are concatenated and represented as the
style information of $x_{t}$. Let
$s(x_{t})={H^{l}(x_{t})}^{\frown}{H^{a}(x_{t})}^{\frown}{H^{b}(x_{t})}$ denote
the concatenated histograms of $x_{t}$, and we take $s(x_{t})$ as input to ADS
for domain separation. However, most existing clustering algorithms require a
hyperparameter to determine the number of clusters. Directly applying a naive
clustering might lead to a nonoptimal adaptation performance. Thus, we propose
to find the optimal number $k^{*}$ of the subdomains using Silhouette
Coefficient (SC) [23]. Suppose the target domain $\mathcal{T}$ is separated
into $k$ subdomains, $\\{T_{1},\dots,T_{k}\\}$. For each target image $x_{t}$,
we denote $\gamma(x_{t})$ as the average distance between $x_{t}$ and all
other target images in the target subdomain to which $x_{t}$ belongs.
Additionally, we use $\delta(x_{t})$ to represent the minimum average distance
from $x_{t}$ to all other target subdomains to which $x_{t}$ does not belong.
Let us assume $x_{t}$ belongs to the $m\textsuperscript{th}$ target subdomain
$T_{m}$, then $\gamma(x_{t})$ and $\delta(x_{t})$ are written as

$\displaystyle\gamma(x_{t})$ $\displaystyle=\frac{\sum_{{x_{t}}^{\prime}\in
T_{m},{x_{t}}^{\prime}\not=x_{t}}L(s({x_{t}}^{\prime}),s(x_{t}))}{|T_{m}|-1},$
(1) $\displaystyle\delta(x_{t})$ $\displaystyle=\min_{T_{n}:1\leq n\leq
k,n\not=m}\left\\{\frac{\sum_{{x_{t}}^{\prime}\in
T_{n}}L(s({x_{t}}^{\prime}),s(x_{t}))}{|T_{n}|}\right\\},$

where $L(s({x_{t}}^{\prime}),s(x_{t}))$ represents the euclidean distance of
$s({x_{t}}^{\prime})$ and $s(x_{t})$, and $|T_{m}|$ is the number of the
target images in $T_{m}$. The SC score for $k$ number of the target subdomains
is given by

$SC(k)=\sum_{x_{t}\in\mathcal{T}}{\frac{\delta(x_{t})-\gamma(x_{t})}{\max(\gamma(x_{t}),\delta(x_{t}))}}.$
(2)

Hence, the goal of the proposed ADS is to find $k^{*}$ for

$k^{*}=\arg\max_{k}SC(k).$ (3)

###  3.2 Subdomain Style Purification

With the help fo automatic domain separation, the number of abnormal samples
with different styles is small inside each target subdomain. Though these
abnormal samples might be useful for the model’s generalization, they could
also lead to a negative transfer, which further hinders the model from
learning domain invariant features in a specific subdomain. To cope with it,
we propose to purify the style distribution of the target images inside each
subdomain. We design a subdomain style purification (SSP) module to
effectively make similar styles for the images within the same subdomain.
Given the $m\textsuperscript{th}$ target subdomain $T_{m}$, we adopt the
histograms of LAB color space
$\\{(H^{l}(x_{t}),H^{a}(x_{t}),H^{b}(x_{t}));\forall x_{t}\in T_{m}\\}$
(mentioned in 3.1), and then we compute the mean of the histograms for all the
three channels, represented by $\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a}$,
and $\widetilde{H}_{m}^{b}$, and this process is achieved by

$\widetilde{H}_{m}^{c}=\frac{\sum_{x_{t}\in
T_{m}}H^{c}(x_{t})}{|T_{m}|};\forall c\in\\{l,a,b\\},$ (4)

where $|T_{m}|$ represents the number of the target images in $T_{m}$. We take
$\\{\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a},\widetilde{H}_{m}^{b}\\}$ as
the standard style for $T_{m}$. For each target RGB image $x_{t}\in T_{m}$, we
change the style of $x_{t}$ to generate the RGB new image $\tilde{x}_{t}$ by
the histogram matching [20] on $\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a}$,
and $\widetilde{H}_{m}^{b}$ on the LAB color space. The process of SSP is done
for all the subdomains $\\{T_{1},\dots,T_{k^{*}}\\}$. We denote the purified
subdomains after SSP as $\\{\widetilde{T}_{1},\dots,\widetilde{T}_{k^{*}}\\}$.

##  4 Multi-teacher Framework

Figure 2: (a) The architecture of the proposed bidirectional photometric
mixing. (b) The diagram of the multi-teacher learning framework.

###  4.1 Bidirectional Photometric Mixing

Through automatic domain separation and subdomain style purification
(mentioned in 3.1 and 3.2), the compound domain $\mathcal{T}$ is automatically
separated into multiple subdomains
$\\{\widetilde{T}_{1},\dots,\widetilde{T}_{k^{*}}\\}$, where $k^{*}$
represents the optimal number of the subdomains. Our next plan is to minimize
the domain gap between the source domain and each target subdomain. A recent
UDA work DACS [25] presents a mixing-based UDA technique for semantic
segmentation. Inspired by DACS, we propose bidirectional photometric mixing
(BPM) to minimize the domain gap between the source domain and each target
subdomain separately. Compared with DACS, the proposed BPM adopts a
photometric transform to decrease the style inconsistency of the mixed images
to reduce the pixel-level domain gap. On this basis, BPM applies a
bidirectional mixing scheme to provide a more robust regularization for
training. The architecture of BPM is shown in Figure 2(a). The proposed BPM
contains a domain adaptive segmentation network $G_{m}$ and a momentum network
$M_{m}$ that improves the stability of pseudo labels. Let
$(x_{s},y_{s})\in\mathcal{S}$ denote the source RGB image and its pixel-wise
annotation map, $x_{s}\in\mathbb{R}^{H\times W\times 3}$,
$y_{s}\in\mathbb{R}^{H\times W}$. And $(\tilde{x}_{t})\in\widetilde{T}_{m}$
represent a purified target RGB image from the $m\textsuperscript{th}$
purified subdomain $\widetilde{T}_{m}$, $\tilde{x}_{t}\in\mathbb{R}^{H\times
W\times 3}$. Note that $H$ and $W$ represent the size of height and width. Our
BPM applies the mixing in two directions: $\mathcal{S}\to\widetilde{T}_{m}$
and $\widetilde{T}_{m}\to\mathcal{S}$.

On the direction of mixing from $\mathcal{S}\to\widetilde{T}_{m}$, we choose
ClassMix [15] because the source image $x_{s}$ has the pixel-wise annotation
map $y_{s}$. We first randomly select some classes from $y_{s}$. Then, we
define $\Psi\in\\{0,1\\}^{H\times W}$ as a binary mask in which $\Psi(h,w)=1$
when the pixel position $(h,w)$ of $x_{s}$ belongs to the selected classes,
and $\Psi(h,w)=0$ otherwise. While ClassMix suggests directly copying the
corresponding pixels of selected classes of $x_{s}$ onto $\tilde{x}_{t}$, the
mixed image generated by ClassMix contains inconsistent style distribution
which might hinder the adaptation performance. To cope with the limitation,
the proposed BPM applies photometric transform $\Gamma$ on the selected source
pixels to the style of target image before directly copying them onto it. Let
$\Psi\odot x_{s}$ represent the selected source pixels by the mask $\Psi$, and
$\odot$ is element-wise multiplication. We first calculate the histograms of
selected source pixels in LAB color space, and match them with
$\\{\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a},\widetilde{H}_{m}^{b}\\}$. The
translated source pixels is represented as $\Gamma(\Psi\odot x_{s})$. Then, we
copy the translated source pixels onto $\tilde{x}_{t}$. We present some
qualitative results in Figure 4. Note that no ground-truth annotation is
available for $\tilde{x}_{t}$. Thus, we send the purified target image
$\tilde{x}_{t}$ to the momentum network $M_{m}$ to generate a stable
prediction map $\tilde{y}^{\prime}_{t}$ as the pseudo label. The mixing
process on the direction of $\mathcal{S}\to\widetilde{T}_{m}$ by BPM is shown
as

$\displaystyle x_{\psi}$ $\displaystyle=\Gamma(\Psi\odot
x_{s})+(\mathbf{1}-\Psi)\odot\tilde{x}_{t},$ (5) $\displaystyle y_{\psi}$
$\displaystyle=\Psi\odot y_{s}+(\mathbf{1}-\Psi)\odot\tilde{y}^{\prime}_{t},$

where $x_{\psi}$ is the generated mixed image, $y_{\psi}$ is the corresponding
mixed pseudo label, and $\Gamma(\cdot)$ is the photometric transform of the
source selected pixels by histogram matching on LAB color space.

On the direction of mixing from $\widetilde{T}_{m}\to\mathcal{S}$, however, it
is impossible to choose ClassMix since no ground-truth annotation is available
for $\tilde{x}_{t}$. Inspired by CutMix [31], we generate another binary mask
$\Phi\in\\{0,1\\}^{H\times W}$ by sampling rectangular bounding box
$(d_{x},d_{y},d_{w},d_{h})$ according to the uniform distribution; $d_{x}\sim
U(0,W),d_{y}\sim U(0,H),d_{w}=W\sqrt{1-\eta},d_{h}=H\sqrt{1-\eta}$, where
$\eta\sim U(0,1)$, $(H,W)$ are the height and width of the image. The binary
mask $\Phi$ is formed by filling with $1$ the pixel positions inside the
bounding box, and filling with $0$ other positions. With the help of $\Phi$,
we select the target pixels $\Phi\odot\tilde{x}_{t}$ and transform them into
the source style. The transformed target pixel is represented by
$\Delta(\Phi\odot\tilde{x}_{t})$. Then we paste them onto the source image
$x_{s}$. We present the mixing of $\widetilde{T}_{m}\to\mathcal{S}$ at

$\displaystyle x_{\phi}$
$\displaystyle=\Delta(\Phi\odot\tilde{x}_{t})+(\mathbf{1}-\Phi)\odot x_{s},$
(6) $\displaystyle y_{\phi}$
$\displaystyle=\Phi\odot\tilde{y}^{\prime}_{t}+(\mathbf{1}-\Phi)\odot y_{s},$

where $x_{\phi}$ is the other generated mixed image, $y_{\phi}$ is the
corresponding mixed pseudo label, and $\Delta(\cdot)$ is the photometric
transform of the target selected pixels by histogram matching on LAB color
space.

we $(x_{\psi},y_{\psi})$ $(x_{\phi},y_{\phi})$ and $(x_{s},y_{s})$ to train
the segmentation network $G_{m}$ and the momentum network $M_{m}$. We first
optimize the parameters of $G_{m}$ through

$\begin{aligned} \mathcal{L}_{BGM}(\theta_{m})=\sum_{\forall x_{s}\in
S}\sum_{\forall\tilde{x}_{t}\in\widetilde{T}_{m}}\Big{[}\mathcal{L}_{CE}\Big{(}G_{m}(x_{s}),y_{s}\Big{)}&+\alpha\mathcal{L}_{CE}\Big{(}G_{m}(x_{\psi}),y_{\psi}\Big{)}\\\
&+\beta\mathcal{L}_{CE}\Big{(}G_{m}(x_{\phi}),y_{\phi}\Big{)}\Big{]}\end{aligned},$
(7)

where $\theta_{m}$ represent the parameters of $G_{m}$, $\mathcal{L}_{CE}$ is
the cross-entropy loss for the predicted segmentation maps and the ground-
truth or pseudo labels, $\alpha$ and $\beta$ are the hyper-parameters to
control the effect of the mixing of both the directions for the loss function.
To help the momentum network $M_{m}$ provide stable pseudo labels, we update
the parameters of $M_{m}$, represented by ${\theta}^{\prime}_{m}$, using an
exponential moving average (EMA) with a momentum $\lambda\in[0,1]$. After
finishing the training iteration $t$, ${\theta}^{\prime}_{m}$ is updated by

${\theta^{\prime}_{m}}^{t+1}=\lambda{\theta^{\prime}_{m}}^{t}+(1-\lambda)\theta_{m}.$
(8)

###  4.2 Multi-teacher Adaptive Knowledge Distillation

We propose a multi-teacher framework followed by an adaptive knowledge
distillation to align the domain gaps between the source domain and all the
target subdomains. Given a purified subdomain $\widetilde{T}_{m}$, we adopt a
BPM as a specific teacher model to minimize the domain gap between
$\mathcal{S}$ and $\widetilde{T}_{m}$. And we train the proposed multi-teacher
framework by minimizing the loss function $\mathcal{L}_{MT}$ on all the
teacher models, i.e.,

$\mathcal{L}_{MT}=\sum_{m=1}^{k^{*}}\mathcal{L}_{BGM}(\theta_{m}),$ (9)

where $\mathcal{L}_{BGM}(\theta_{m})$ (defined in Equation 7) is the loss
function of the segmentation network $G_{m}$ in the $m\textsuperscript{th}$
teacher model, and $k^{*}$ is the optimal number of the subdomains. Moreover,
We learn a segmentation network $G_{sd}$ as the student network via an
adaptive knowledge distillation from all the teacher networks $\\{G_{m}:1\leq
m\leq k^{*}\\}$. Given a random target data from $x_{t}\in\mathcal{T}$, we
send $x_{t}$ to all the teachers model, and the student is to learn from a
weighted average of the all teacher’s predictions $O_{w}(x_{t})$, based on the
teacher’s confidence score. We adopt the entropy of $G_{m}$’s prediction map
$G_{m}(x_{t})\in\mathbb{R}^{H\times W\times C}$ as the confidence of the
$m\textsuperscript{th}$ teacher model, where $C$ is the total number of
classes we consider. Thus, the weight $w_{m}$ for the $m\textsuperscript{th}$
teacher and the average prediction $G_{out}(x_{t})$ are formulated as

$\displaystyle
w_{m}=\frac{\sum_{h,w,c}G_{m}(x_{t})\log\big{[}G_{m}(x_{t})\big{]}}{\sum_{m^{\prime}}\sum_{h,w,c}G_{m^{\prime}}(x_{t})\log\big{[}G_{m^{\prime}}(x_{t})\big{]}},$
(10) $\displaystyle G_{out}(x_{t})=\sum_{m=1}^{k^{*}}w_{m}G_{m}(x_{t}).$

On this basis, we optimize the student segmentation network $G_{sd}$ with a
distillation loss $\mathcal{L}_{D}$ defined by

$\mathcal{L}_{D}=\sum_{x_{t}\in\mathcal{T}}\mathcal{L}_{KL}\Big{[}G_{sd}(x_{t})||G_{out}(x_{t})\Big{]},$
(11)

where $\mathcal{L}_{KL}$ is KL divergence loss function between the output of
$G_{sd}$ and $G_{out}$. The goal of the multi-teacher adaptive knowledge
distillation is to achieve the optimal parameters ${\theta_{sd}}^{*}$ of the
student segmentation network $G_{sd}$ by

${\theta_{sd}}^{*}=\min_{\theta_{sd}}\mathcal{L}_{MT}+\mathcal{L}_{D}.$ (12)

Online Updating with Consistency Regularization. To evaluate the
generalization of our approach, we directly evaluate our student network on
the open domains as shown in Table 2(a) and Table 2(b). Additionally, after
finishing the compound domain adaptation training, we also provide a fast and
short online updating for the student network using consistency
regularization. This would further boost the generalization of the student
network. Given an RGB image $x_{o}$ from an open domain, we first match the
style of $x_{o}$ to other standard styles from the existing target subdomains.
The standard styles are defined as the mean histograms
$\\{\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a},\widetilde{H}_{m}^{b}\\}$
(defined in 3.2). The newly transformed images are $\\{x_{o}^{m};1\leq m\leq
k^{*}\\}$, where $x_{o}^{m}$ is generated by matching $x_{o}$ to the style of
the $m\textsuperscript{th}$ subdomain $\widetilde{T}_{m}$. Thus, we conduct an
online updating for the student network $G_{sd}$ by

$\min_{\theta_{sd}}\sum_{m=1}^{k^{*}}\mathcal{L}_{1}\big{(}G_{sd}(x_{o}^{m}),G_{sd}(x_{o})\big{)},$
(13)

where $\mathcal{L}_{1}$ is the mean absolute loss. After the online updating,
we test the student network with newly learnt parameters again on the open
domains.

##  5 Experiments

###  5.1 Experimental Setup

####  5.1.1 Dataset.

In this work, we adopt the synthetic datasets, including GTA5 [21] and SYNTHIA
[22] as the source domains. GTA5 contains $24,966$ annotated images of
$1,914\times 1,052$ resolution. SYNTHIA consists of $9,400$ images with
$1,280\times 760$ resolution. Furthermore, we adopt C-Driving [13] as the
compound target domains which contains real images of $1,280\times 720$
resolution collected from different weather conditions. Following the settings
of previous works [13, 19, 8], we use the $14,697$ rainy, snowy, cloudy images
as the compound target domain and adopt $627$ overcast images as the open
domain. We also use ACDC [24] as another compound target domain and the
evaluation results are shown in supplementary material. We further adopt
Cityscapes [5], KITTI [1], and WildDash [32] as the open domains to evaluate
the generalization ability of the proposed approach.

####  5.1.2 Implementation Details.

We adopt DeepLab-V2 [3] with ResNet101 backbone [9] pre-trained on ImageNet
[6]. All the images from target domain are rescaled into $1,280\times 720$ and
then randomly cropped into $640\times 360$. The batch size is set up with $2$
and the total number of training iterations is $2.5\times 10^{5}$. We adopt
stochastic gradient descent to optimize all the segmentation networks, with a
weight decay of $5\times 10^{-4}$ and momentum of $0.9$. The learning rate is
set up with an initial value of $2.5\times 10^{-4}$ and decreased by
polynomial decay with an exponent of $0.9$. The momentum network has the same
network architecture as the segmentation network. Existing mixing techniques
contain CutMix [31], CowMix [7] and ClassMix [15]. We adopt ClassMix on the
mixing direction of the source domain to the target domain, and we apply
CutMix on the mixing direction of the target domain to the source domain. Both
$\alpha$ and $\beta$ are set up with $1$ in the experiments. To increase the
robustness of the segmentation model, we adopt data augmentations, including
flipping, color jittering, and Gaussian blurring on the mixed images.

Table 1: The performance comparison of mean IoU on the compound domain. Our
approach is compared with the state-of-the-art UDA and OCDA approaches on (a)
GTA5$\rightarrow$C-Driving and (b) SYNTHIA$\rightarrow$C-Driving benchmark
dataset with ResNet-101 as the backbone. Note that mIoU11 represents the mean
IoU of 11 classes, excluding the class with ∗.

(a) GTA5$\to$C-Driving |
---|---
Method | Type |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  mbike |  bike | mIoU
Source | - | 73.4 | 12.5 | 62.8 | 6.0 | 15.8 | 19.4 | 10.9 | 21.1 | 54.6 | 13.9 | 76.7 | 34.5 | 12.4 | 68.1 | 31.0 | 12.8 | 0.0 | 10.1 | 1.9 | 28.3
CDAS [13] | OCDA | 79.1 | 9.4 | 67.2 | 12.3 | 15.0 | 20.1 | 14.8 | 23.8 | 65.0 | 22.9 | 82.6 | 40.4 | 7.2 | 73.0 | 27.1 | 18.3 | 0.0 | 16.1 | 1.5 | 31.4
CSFU [8] | OCDA | 80.1 | 12.2 | 70.8 | 9.4 | 24.5 | 22.8 | 19.1 | 30.3 | 68.5 | 28.9 | 82.7 | 47.0 | 16.4 | 79.9 | 36.6 | 18.8 | 0.0 | 13.5 | 1.4 | 34.9
SAC [2] | UDA | 81.5 | 23.8 | 72.0 | 10.3 | 27.8 | 23.0 | 18.2 | 34.1 | 70.3 | 27.9 | 87.8 | 45.0 | 16.9 | 77.6 | 38.5 | 19.8 | 0.0 | 14.0 | 2.7 | 36.4
DACS [25] | UDA | 81.9 | 24.0 | 72.2 | 11.9 | 28.6 | 24.2 | 18.3 | 35.4 | 71.8 | 28.0 | 87.7 | 44.9 | 15.6 | 78.4 | 39.1 | 24.9 | 0.1 | 6.9 | 1.9 | 36.6
DHA[19] | OCDA | 79.9 | 14.5 | 71.4 | 13.1 | 32.0 | 27.1 | 20.7 | 35.3 | 70.5 | 27.5 | 86.4 | 47.3 | 23.3 | 77.6 | 44.0 | 18.0 | 0.1 | 13.7 | 2.5 | 37.1
Ours | OCDA | 85.3 | 26.2 | 72.8 | 10.6 | 33.1 | 26.9 | 24.6 | 39.4 | 70.8 | 32.5 | 87.9 | 47.6 | 29.2 | 84.8 | 46.0 | 22.8 | 0.2 | 16.7 | 5.8 | 40.2

(b) SYNTHIA$\to$C-Driving |
---|---
Method | Type |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign∗ |  veg |  sky |  person |  rider∗ |  car |  bus∗ |  mbike∗ |  bike∗ | mIoU16 | mIoU11
Source | - | 33.9 | 11.9 | 42.5 | 1.5 | 0.0 | 14.7 | 0.0 | 1.3 | 56.8 | 76.5 | 13.3 | 7.4 | 57.8 | 12.5 | 2.1 | 1.6 | 20.9 | 28.1
CDAS [13] | OCDA | 54.5 | 13.0 | 53.9 | 0.8 | 0.0 | 18.2 | 13.0 | 13.2 | 60.0 | 78.9 | 17.6 | 3.1 | 64.2 | 12.2 | 2.1 | 1.5 | 25.3 | 34.0
CSFU [8] | OCDA | 69.6 | 12.2 | 50.9 | 1.3 | 0.0 | 16.7 | 12.1 | 13.6 | 56.2 | 75.8 | 20.0 | 4.8 | 68.2 | 14.1 | 0.9 | 1.2 | 26.1 | 34.8
SAC [2] | UDA | 69.8 | 13.4 | 56.2 | 1.7 | 0.0 | 20.0 | 9.6 | 13.7 | 52.5 | 78.1 | 29.1 | 15.5 | 68.9 | 10.9 | 3.2 | 1.2 | 27.7 | 36.3
DACS [25] | UDA | 62.1 | 15.2 | 48.8 | 0.3 | 0.0 | 19.7 | 10.3 | 9.6 | 57.8 | 84.4 | 35.2 | 18.9 | 67.8 | 16.0 | 2.2 | 1.7 | 28.1 | 36.5
DHA [19] | OCDA | 67.5 | 2.5 | 54.6 | 0.2 | 0.0 | 25.8 | 13.4 | 27.1 | 58.0 | 83.9 | 36.0 | 6.1 | 71.6 | 28.9 | 2.2 | 1.8 | 29.9 | 37.6
Ours | OCDA | 73.4 | 15.2 | 57.1 | 1.8 | 0.0 | 23.2 | 13.5 | 23.9 | 59.9 | 83.3 | 40.3 | 22.3 | 72.2 | 23.3 | 2.3 | 2.2 | 32.1 | 40.0

###  5.2 Results

To demonstrate the efficacy of our approach, we conduct experiments on the
benchmark datasets of GTA5$\to$C-Driving and SYNTHIA$\to$C-Driving. We first
compare our approach with the existing state-of-the-art OCDA approaches: CDAS
[13], DHA [19], and CSFU [8]. Furthermore, we compare the proposed approach
with the current state-of-the-art UDA approaches SAC [2] and DACS [25].

####  5.2.1 Compound Domain Adaptation.

We first compare the performance of our approach with existing state-of-the-
art OCDA and UDA approaches on GTA5 $\rightarrow$C-Driving, shown in Table 1a.
All the results are generated on the validation set of C-Driving. Training
only with the source data leads to $28.3\%$ of mean IoU over the 19 classes.
As the first work in OCDA, CDAS achieves $31.4\%$ on the mean IoU of all the
classes. CSFU generates $34.9\%$ of mean IoU, and DHA produces $37.1\%$ of
mean IoU. This is because both CSFU and DHA adopt the subdomain separation
step and GAN framework, and DHA uses a more effective multi-discriminator to
minimize the domain gaps. In comparison, the latest UDA approaches DACS and
SAC show $36.6\%$ and $36.4\%$, outperforming both CDAS and CSFU. The reason
behind is that both DACS and SAC adopt various self-supervision techniques to
minimize the domain gaps, which proves to be more effective than GAN-based
approaches. In comparison, the proposed approach demonstrates effectiveness on
this benchmark dataset with $40.2\%$ of mean IoU over all classes.

We present experimental results on SYNTHIA$\rightarrow$C-Driving shown in
Table 1b. We consider the 11 classes for final evaluation. The proposed method
achieves $40.0\%$ of mean IoU over the 11 classes. For other OCDA approaches,
DHA achieves $37.6\%$, CSFU produces $34.8\%$, and CDAS generates $34.0\%$ of
mean IoU. Moreover, the UDA approaches DACS and SAC generate $36.5\%$ and
$36.3\%$ of mean IoU. Our approach outperforms all the existing OCDA
approaches and the latest UDA approaches.

Table 2: The comparison of mean IoU on the open domains. The domain
generalization (DG) model is trained only with the source domain. All the
models are tested on the validation set of C-Driving Open (O), cityscapes (C),
KITTI (K), and wildDash (W). We also present the scores of our approach
without online updating (w/o Updating) and with online updating (w/ Updating).

(a) GTA5 as the source domain.

GTA5
---
Method | Type | O | C | K | W | Avg
CSFU [8] | OCDA | 38.9 | 38.6 | 37.9 | 29.1 | 36.1
DACS [25] | UDA | 39.7 | 37.0 | 40.2 | 30.7 | 36.9
RobustNet [4] | DG | 38.1 | 38.3 | 40.5 | 30.8 | 37.0
DHC [19] | OCDA | 39.4 | 38.8 | 40.1 | 30.9 | 37.5
Ours (w/o Updating) | OCDA | 41.8 | 40.9 | 44.0 | 32.9 | 40.0
Ours (w/ Updating) | OCDA | 42.5 | 41.7 | 44.3 | 34.6 | 40.8

(b) SYNTHIA as the source domain.

SYNTHIA
---
Method | Type | O | C | K | W | Avg
CSFU [8] | OCDA | 36.2 | 34.9 | 32.4 | 27.6 | 32.8
DACS [25] | UDA | 36.8 | 37.0 | 37.4 | 28.8 | 35.0
RobustNet [4] | DG | 37.1 | 38.3 | 40.1 | 29.6 | 36.3
DHC [19] | OCDA | 38.9 | 38.0 | 40.6 | 30.0 | 36.9
Ours (w/o Updating) | OCDA | 41.5 | 40.3 | 42.7 | 30.1 | 38.7
Ours (w/ Updating) | OCDA | 42.6 | 41.1 | 43.4 | 30.9 | 39.5

####  5.2.2 Generalization to the Open Domains.

We also evaluate the domain generalization of the proposed approach against
existing UDA and OCDA approaches. The results are presented in Table 2(a) and
2(b). Our work is compared with the latest domain generalization (DG) approach
RobustNet [4]. For all the UDA and OCDA approaches, we first train them with
the labeled source and the unlabeled target images, and we evaluate their
performance with the validation of the open domains. RobustNet generates
$37.0\%$ of mean IoU in Table 2(a) and $36.3\%$ of mean IoU in Table 2(b).
Note that RobustNet only requires labeled source data during training. This
shows that the DG approach is more effective in generalizing to the open
domains than the existing UDA and OCDA approaches DACS and CSFU. Without any
online updating, our approach achieves $40.0\%$ of mean IoU in Table 2(a) and
$38.7\%$ of mean IoU in Table 2(b). Our approach outperforms all the UDA
approaches, OCDA approaches, and the DG approach listed in the table. The
reason might be that our approach is more powerful for learning the domain
invariant features which improve the generalization of the model toward novel
domains. The performance gain of our approach with updating further shows the
efficacy of the proposed online updating with consistency regularization.

###  5.3 Ablation Study

####  5.3.1 Generating Optimal Subdomains.

Figure 3: We conduct the ablation study on the proposed automatic domain
separation using GTA5$\to$C-Driving with ResNet101 backbone. (a) The
scatterplot shows the correlation between our approach’s mean IoU and the
Silhouette Coefficient score. (b) The mean IoU of our approach with different
number of subdomains $k$. (c) The sample images from the subdomains of the
C-Driving dataset.

We first conduct the ablation study on the correlation between the mean IoU of
the proposed approach with Silhouette Coefficient (SC) score on the subdomain
separation in Figure 3(a). It shows a positive correlation, which means that
the SC score is effectively finds the optimal number of subdomains for the
compound target domain. Moreover, we evaluate the mean IoU score with the
different number of subdomains $k$ in Figure 3(b). Finally, we set up $k=3$
and present the sample images from the subdomains of the C-Driving dataset in
Figure 3(c). We also evaluate the efficacy of the proposed subdomain style
purification (SSP) in Table 3(b). Without using SSP, the performance drops
$0.5\%$ of mean IoU.

Table 3: The ablation study on the efficacy of the components of our model.
(a) We compare with one baseline model DACS [25] and evaluate the performance
gain of the bidirectional photometric mixing and the multi-teacher learning.
(b) We evaluate the performance drop of our model by removing each component
from it. Our model is trained GTA5$\to$C-Driving with ResNet101 backbone and
tested on C-Driving validation set.

(a) The performance gain.

GTA5$\rightarrow$C-Driving
---
Model | mIoU
DACS [25] | 36.6
DACS + Multi-teacher Learning | 39.1
DACS + Bidirectional Mixing | 37.3
DACS + Photometric Mixing ($\Gamma,\Delta$) | 37.4
DACS + Bidirectional Photometric Mixing | 37.8
Ours | 40.2

(b) The performance drop.

GTA5$\rightarrow$C-Driving
---
Configuration | mIoU | Gap
w/o Multi-teacher Learning | 38.0 | -2.2
w/o Mixing on One Direction ($\alpha=0$) | 38.5 | -1.7
w/o Mixing on One Direction ($\beta=0$) | 38.9 | -1.3
w/o Subdomain Style Purification | 39.7 | -0.5
w/o Adaptive Distillation | 39.6 | -0.6
Full Framework | 40.2 | -

####  5.3.2 Multi-teacher and Single Model.

The ablation study on the multi-teacher learning of our proposed approach is
presented in Table 3(a) and Table 3(b). Applying a single model in our
approach delivers $38.0\%$ of mean IoU, leading to the the most significant
drop $2.2\%$, shown in Table 3(b). We further combine DACS with multi-teacher
learning, and the mean IoU reaches from $36.6\%$ to $39.1\%$. We argue that
utilizing a single model is less effective than the multi-teacher models.
Because adapting to one subdomain might hinder the single model from adapting
to other dissimilar subdomains. Thus, we employ a multi-teacher framework in
which each teacher adapts to one subdomain separately. And the multiple
teachers together provide a comprehensive guide to the student model to adapt
to all the target subdomains. We further present the qualitative results about
the target image prediction maps from each subdomain by the multi-teachers and
the single-teacher model in Figure 5.

Figure 4: We compare the mixed images from the source domain to the target
domain. (a) the source image; (b) the target image; (c) the mixed images
without using photometric transform, and the style inconsistency exists; (d)
the mixed images using photometric transform, and the style inconsistency is
mitigated; (e) the mask to crop the source image.

####  5.3.3 Bidirectional Photometric Mixing.

We further conduct the ablation study for the bidirectional photometric mixing
(BPM), shown in Table 3(a) and Table 3(b). Our model is trained on GTA5 $\to$
C-Driving with ResNet101 backbone and tested on C-Driving validation set. By
making $\alpha=0$ to remove the mixing on one direction (ClassMix), the mean
IoU drops $1.7\%$, while making $\beta=0$ to remove the other directional
mixing (CutMix), it decreases by $1.3\%$. This suggests that ClassMix
contributes slightly more to the final performance. We also use the baseline
model DACS for an in-depth analysis. We add the bidirectional photometric
mixing with the DACS, the performance increase from $36.6\%$ to $37.8\%$ shown
in Table 3(a); we then combine DACS with only bidirectional mixing, the mean
IoU rise up to $37.3\%$; we further add DACS with only photometric transform
on mixing (use $\Gamma$ and $\Delta$), the mean IoU reaches to $37.4\%$. The
reason behind is that DACS utilizes a simple mixing method that contains only
one direction and generates the mixed image with the style inconsistency
inside. However, we propose a bidirectional mixing scheme and apply the
photometric transform to mitigate the style inconsistency on the generated
images. We present the qualitative results to show this issue in Figure 4. The
style inconsistency is mitigated in Figure 4(d) compared with Figure 4(c) on
the mixing direction from the source domain to the target domain.

Figure 5: We present the predicted segmentation maps of the target images from
every target subdomain. The maps in the second row are generated using a
single model. The maps in the third row are generated using the multi-teacher
models.

##  6 Conclusion

Open compound domain adaptation (OCDA) considers the target domain as the
compound of multiple unknown subdomains. In this work, we first propose
automatic domain separation to find the optimal number of subdomains. Then we
design a multi-teacher framework with bidirectional photometric mixing to
align the domain gap between the source domain and the compound target domain,
and we further evaluate its generalization to novel domains. Our current work
is only focused on segmentation task and we leave the study on other visual
tasks for future research.

##  7 Acknowledgment

This work was supported by the Korea Institute of Energy Technology Evaluation
and Planning (KETEP) and the Ministry of Trade, Industry & Energy (MOTIE) of
the Republic of Korea (No. 20224000000100).

## References

  * [1] Abu Alhaija, H., Mustikovela, S.K., Mescheder, L., Geiger, A., Rother, C.: Augmented reality meets computer vision: Efficient data generation for urban driving scenes. IJCV 126(9), 961–972 (2018)
  * [2] Araslanov, N., Roth, S.: Self-supervised augmentation consistency for adapting semantic segmentation. In: CVPR. pp. 15384–15394 (2021)
  * [3] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. PAMI 40(4), 834–848 (2017)
  * [4] Choi, S., Jung, S., Yun, H., Kim, J.T., Kim, S., Choo, J.: Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In: CVPR. pp. 11580–11590 (2021)
  * [5] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR. pp. 3213–3223 (2016)
  * [6] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. IEEE (2009)
  * [7] French, G., Oliver, A., Salimans, T.: Milking cowmask for semi-supervised image classification. arXiv preprint arXiv:2003.12022 (2020)
  * [8] Gong, R., Chen, Y., Paudel, D.P., Li, Y., Chhatkuli, A., Li, W., Dai, D., Van Gool, L.: Cluster, split, fuse, and update: Meta-learning for open compound domain adaptive semantic segmentation. In: CVPR. pp. 8344–8354 (2021)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
  * [10] Hoffman, J., Tzeng, E., Park, T., Zhu, J.Y., Isola, P., Saenko, K., Efros, A., Darrell, T.: Cycada: Cycle-consistent adversarial domain adaptation. In: ICML. pp. 1989–1998. PMLR (2018)
  * [11] Huang, J., Guan, D., Xiao, A., Lu, S.: Fsdr: Frequency space domain randomization for domain generalization. In: CVPR. pp. 6891–6902 (2021)
  * [12] Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W.: Ccnet: Criss-cross attention for semantic segmentation. In: CVPR. pp. 603–612 (2019)
  * [13] Liu, Z., Miao, Z., Pan, X., Zhan, X., Lin, D., Yu, S.X., Gong, B.: Open compound domain adaptation. In: CVPR. pp. 12406–12415 (2020)
  * [14] Ma, H., Lin, X., Wu, Z., Yu, Y.: Coarse-to-fine domain adaptive semantic segmentation with photometric alignment and category-center regularization. In: CVPR. pp. 4051–4060 (2021)
  * [15] Olsson, V., Tranheden, W., Pinto, J., Svensson, L.: Classmix: Segmentation-based data augmentation for semi-supervised learning. In: WACV. pp. 1369–1378 (2021)
  * [16] Ouyang, C., Biffi, C., Chen, C., Kart, T., Qiu, H., Rueckert, D.: Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. In: ECCV. pp. 762–780. Springer (2020)
  * [17] Pan, F., Shin, I., Rameau, F., Lee, S., Kweon, I.S.: Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In: CVPR. pp. 3764–3773 (2020)
  * [18] Pan, X., Luo, P., Shi, J., Tang, X.: Two at once: Enhancing learning and generalization capacities via ibn-net. In: ECCV. pp. 464–479 (2018)
  * [19] Park, K., Woo, S., Shin, I., Kweon, I.S.: Discover, hallucinate, and adapt: Open compound domain adaptation for semantic segmentation. In: NeurIPS (2020)
  * [20] Rafael, C.G., Richard, E.W., Steven, L.E., Woods, R., Eddins, S.: Digital image processing using MATLAB. Tata McGraw-Hill (2010)
  * [21] Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: ECCV. pp. 102–118. Springer (2016)
  * [22] Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.: The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: CVPR (2016)
  * [23] Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. JCAM 20, 53–65 (1987)
  * [24] Sakaridis, C., Dai, D., Van Gool, L.: Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In: ICCV. pp. 10765–10775 (2021)
  * [25] Tranheden, W., Olsson, V., Pinto, J., Svensson, L.: Dacs: Domain adaptation via cross-domain mixed sampling. In: WACV. pp. 1379–1389 (2021)
  * [26] Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: CVPR (2018)
  * [27] Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: CVPR. pp. 7472–7481 (2018)
  * [28] Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: CVPR (2019)
  * [29] Wang, Z., Yu, M., Wei, Y., Feris, R., Xiong, J., Hwu, W.m., Huang, T.S., Shi, H.: Differential treatment for stuff and things: A simple unsupervised domain adaptation method for semantic segmentation. In: CVPR. pp. 12635–12644 (2020)
  * [30] Yue, X., Zhang, Y., Zhao, S., Sangiovanni-Vincentelli, A., Keutzer, K., Gong, B.: Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In: ICCV. pp. 2100–2110 (2019)
  * [31] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: CVPR. pp. 6023–6032 (2019)
  * [32] Zendel, O., Honauer, K., Murschitz, M., Steininger, D., Dominguez, G.F.: Wilddash-creating hazard-aware benchmarks. In: ECCV. pp. 402–416 (2018)
  * [33] Zhang, P., Zhang, B., Zhang, T., Chen, D., Wang, Y., Wen, F.: Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. CVPR (2021)
  * [34] Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: CVPR. pp. 6848–6856 (2018)
  * [35] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR. pp. 2881–2890 (2017)
  * [36] Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: CVPR. pp. 6881–6890 (2021)
  * [37] Zou, Y., Yu, Z., Kumar, B., Wang, J.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: ECCV. pp. 289–305 (2018)

Supplemental Material:
ML-BPM: Multi-teacher Learning with Bidirectional Photometric Mixing for Open
Compound Domain Adaptation in Semantic Segmentation

Anonymous ECCV submission

Fei Pan Sungsu Hur Seokju Lee Junsik Kim In So Kweon

##  8 Subdomain Style Purification and the t-SNE Visualization

Figure 6: (a) presents the noisy samples from Subdomain 2 of C-Driving dataset
before subdomain style purification (before SSP) and after subdomain style
purification (after SSP). (b) shows the t-SNE visualization of the
concatenated histograms of the C-Driving dataset on LAB color space when
$k=3$.

As mentioned in Section 3.2, it is hard to guarantee that the images from the
same target subdomain have the same style. In other words, small domain gaps
might still results from the various image styles in each subdomain. We
propose subdomain style purification to unify the styles of the target data
that belongs to the same subdomain so that the domain gaps in these images
could be further reduced. We provide the visualization of the sample images
transformed by subdomain style purification (SSP) from subdomain 2 in Figure 6
(a). Note that the images from _before SSP_ in Figure 6(a) has the styles
different from the _standard style_ , and they are transformed into the
standard style with the help of histogram matching on the LAB color space. We
further set up $k=3$ and present the t-SNE visualization of the concatenated
histograms of the C-Driving images from LAB color space.

The reason of subdomain style purification (SSP). With the help of automatic
domain separation, the number of abnormal samples with different styles is
small. Though these abnormal samples might be helpful for the model’s
generalization, they could also lead to a negative transfer, which further
hinders the model from learning domain invariant features in a specific
subdomain. With GTA5$\to$C-Driving, we get a $0.5\%$ of mIoU drop on average
over all the subdomains without using SSP, as shown in Table 3(b).

##  9 ACDC Dataset

We also evaluate the proposed approach on another ACDC dataset[24]. ACDC
dataset contains real-world images from the road scenes in diverse weather
conditions, including fog, nighttime, rain and snow. We consider the $2,800$
images of fog, nighttime and rain from the training split of ACDC as the
compound domain; the $400$ snow images with pixel-wise annotations of ACDC
training split are taken as the open domain. The final performance is
evaluated on the validation set of ACDC, which contains $306$ images with
ground-truth maps.

Table 4: The performance comparison of mean IoU on the compound target domain
(fog, nighttime, and rain) and the open domain (fog) of ACDC. Our approach is
compared with the state-of-the-art UDA and OCDA approaches on (a)
GTA5$\rightarrow$ACDC and (b) SYNTHIA$\rightarrow$ACDC benchmark dataset with
ResNet-101 as the backbone.

(a) GTA5$\to$ACDC
---
|  | Compound | Open
Method | Type |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  mbike |  bike | mIoU | mIoU
Source | - | 43.6 | 2.5 | 46.2 | 5.2 | 0.1 | 30.3 | 15.3 | 16.3 | 56.9 | 0.0 | 71.5 | 16.3 | 13.7 | 51.4 | 0.0 | 15.1 | 0.0 | 1.4 | 4.2 | 20.5 | 27.1
CDAS[13] | OCDA | 53.2 | 5.9 | 56.1 | 10.1 | 2.6 | 22.0 | 37.1 | 11.4 | 53.9 | 23.5 | 71.3 | 27.6 | 14.6 | 47.5 | 16.8 | 19.5 | 0.0 | 3.2 | 3.8 | 25.3 | 29.1
CSFU[8] | OCDA | 47.0 | 4.1 | 53.0 | 13.9 | 1.0 | 23.2 | 41.2 | 18.8 | 55.8 | 23.2 | 72.1 | 31.5 | 10.8 | 69.1 | 26.4 | 27.8 | 0.2 | 1.7 | 2.6 | 27.6 | 30.5
SAC[2] | UDA | 42.6 | 4.2 | 57.6 | 11.9 | 3.8 | 23.0 | 49.7 | 23.8 | 63.6 | 31.9 | 76.0 | 30.3 | 10.5 | 65.3 | 23.6 | 23.1 | 0.1 | 0.7 | 3.2 | 28.7 | 33.6
DACS[24] | UDA | 48.9 | 9.7 | 54.5 | 16.8 | 5.7 | 22.7 | 42.0 | 22.9 | 61.3 | 29.7 | 73.7 | 32.2 | 11.6 | 63.3 | 23.2 | 26.5 | 0.0 | 1.2 | 5.2 | 29.0 | 34.8
DHA[19] | OCDA | 49.8 | 5.2 | 59.1 | 10.2 | 3.1 | 25.6 | 47.8 | 27.9 | 65.1 | 32.0 | 75.2 | 29.0 | 12.2 | 61.5 | 20.5 | 32.4 | 0.0 | 1.0 | 2.0 | 29.5 | 37.5
Ours | OCDA | 48.4 | 5.0 | 58.2 | 25.3 | 10.0 | 35.1 | 50.4 | 26.7 | 66.8 | 33.3 | 75.8 | 32.1 | 16.7 | 73.5 | 16.8 | 26.6 | 0.2 | 3.9 | 4.6 | 32.1 | 41.6

(b) SYNTHIA$\to$ACDC
---
|  | Compound | Open
Method | Type |  road |  sidewalk |  building |  wall |  fence |  pole |  light |  sign |  veg |  sky |  person |  rider |  car |  bus |  mbike |  bike | mIoU16 | mIoU16
Source | - | 45.2 | 0.2 | 36.7 | 1.7 | 0.6 | 25.7 | 4.0 | 5.6 | 46.6 | 64.3 | 16.9 | 11.3 | 39.6 | 16.5 | 0.6 | 1.9 | 19.8 | 20.5
CDAS[13] | OCDA | 61.3 | 0.7 | 60.1 | 11.7 | 1.8 | 28.4 | 18.8 | 23.5 | 48.6 | 28.9 | 16.5 | 15.9 | 69.2 | 18.4 | 5.4 | 5.6 | 25.9 | 23.3
CSFU[8] | OCDA | 62.6 | 0.3 | 60.3 | 8.6 | 1.8 | 21.3 | 20.7 | 29.1 | 44.5 | 22.1 | 34.5 | 19.0 | 71.1 | 23.2 | 4.4 | 4.3 | 26.7 | 24.8
SAC[2] | UDA | 69.8 | 0.4 | 56.2 | 1.7 | 0.0 | 20.0 | 12.6 | 13.7 | 52.5 | 78.1 | 29.1 | 15.5 | 68.9 | 20.9 | 3.2 | 1.2 | 27.7 | 25.4
DACS[24] | UDA | 55.6 | 1.1 | 55.7 | 0.1 | 0.7 | 25.8 | 31.7 | 18.3 | 65.5 | 53.7 | 31.1 | 16.6 | 69.2 | 22.5 | 2.9 | 3.1 | 28.3 | 27.0
DHA[19] | OCDA | 55.5 | 1.1 | 57.2 | 0.7 | 0.8 | 26.6 | 22.7 | 24.6 | 65.8 | 58.4 | 29.6 | 23.9 | 70.8 | 19.5 | 5.4 | 4.2 | 29.2 | 27.3
Ours | OCDA | 66.7 | 1.7 | 62.4 | 10.8 | 1.4 | 30.8 | 23.9 | 29.2 | 62.6 | 69.0 | 31.6 | 14.6 | 71.8 | 22.9 | 6.8 | 4.5 | 31.9 | 29.1

We present the performance comparison of mean IoU in Table 4. For the compound
target domain of ACDC (fog, nighttime, rain), we achieve $32.1\%$ of mean IoU
on GTA5$\to$ACDC and $31.9\%$ of mean IoU on SYNTHAI$\to$ACDC, outperforming
all the UDA and OCDA approaches in the list. We also evaluate the
generalization of our approach compared with other works. After finishing the
compound domain adaptation training, all the models are directly tested on the
open domain of ACDC (snow). Note that the snow images have never been used in
training before. Under the benchmark datasets GTA5$\to$ACDC and
SYNTHIA$\to$ACDC, our approach shows $41.6\%$ and $29.1\%$ of mean IoU. This
demonstrates that our approach has better generalization ability toward novel
domains (snow).

Table 5: The evaluation on GTA5$\to$C-Driving.

(a) ImageNet pre-trained VGG-16 Backbone
---
Method | Compound (C) | Open (O) | Average
Rainy | Snowy | Cloudy | Overcast | C | C+O
CDAS[13] | 23.8 | 25.3 | 29.1 | 31.0 | 26.1 | 27.3
CSFU[8] | 24.5 | 27.5 | 30.1 | 31.4 | 27.7 | 29.4
DACS [24] | 26.8 | 29.2 | 35.1 | 35.9 | 30.4 | 31.8
DHA[19] | 27.1 | 30.4 | 35.5 | 36.1 | 32.0 | 32.3
Ours | 34.5 | 35.8 | 39.9 | 40.1 | 36.7 | 37.5

(b) Mixing Algorithm Comparison
---
Algorithm | BPM (_Ours_) | ClassMix [15] | CutMix [31] | CowMix [7]
mIoU | 40.2 | 39.1 | 37.6 | 37.4

##  10 The Practicability of Our Approach

Though we use the multi-teacher models for training, our approach still has
strong practicability for the two following reasons: these teacher models are
trained simultaneously; only a single student model from distillation is
needed for inference. The size of the student model is not affected by the
number of the subdomains. With the number of the subdomains $k^{*}$, the FLOPS
and the number of parameters of our _multi-teacher’s_ model are $327.08\times
10^{9}$ and $43.8\times k^{*}\times 10^{6}$. After the adaptive knowledge
distillation, the FLOPS and number of parameters of our _student_ model is
$327.08\times 10^{9}$ and $43.8\times 10^{6}$.

The VGG-16 backbone and different mixup algorithms. We use VGG-16 backbone
network for evaluation. The experimental results on GTA5$\to$C-Driving in
Table 5(a) demonstrates the effectiveness of our approach against existing
works with ImageNet pre-trained VGG-16 as the backbone. We provide the
comparison to existing domain mixup algorithms in the same setting, including
ClassMix[15], CutMix[31], and CowMix[7].

The online updating on the open domains. Our online updating is conducted on
each sample from the open domain, thus it is still domain generalization at
the testing stage. Our student model $G_{sd}$ is trained through the adaptive
distillation from all the subdomain’s segmentation models
$\\{G_{m}\\}_{m=1}^{k^{*}}$ (Eq. (10, 11)). Each $G_{m}$ is optimized by Eq.
(7) with the help of the mean teacher $M_{m}$, following the work of DACS[24].
We also used $M_{m}$ instead of $G_{m}$ for distillation but do not see
significant performance gain.

The reason of using bidirectional mixing. The performance of using pseudo-
labels of target data for ClassMix. Using the photometric transform $\Delta$
(Eq.(6)) on target-to-source (t2s) mixing, we enforce the consistency of
prediction between the target and the mixed image, which are taken as
additional augmentation to improve the model’s performance (Table.3(a,b)).
With the experiment on GTA5$\to$C-Driving, we get $40.1\%$ of mIoU on using
pseudo-labels of target data for ClassMix on target-to-source mixing, similar
to ours $40.2\%$ (Table 1(a)). Table 5 (b) shows that our BPM outperforms
existing mixing algorithms ClassMix [15], CutMix [31], and CowMix [7].