# On the Verification and Validation of AI Navigation Algorithms

Ivan Porres1, Sepinoud Azimi1, Sébastien Lafond1, Johan Lilius1, Johanna
Salokannel2 and Mirva Salokorpi2  1Faculty of Science and Engineering
Åbo Akademi University
Turku, Finland
<EMAIL_ADDRESS>2Novia University of Applied Sciences
Turku, Finland
<EMAIL_ADDRESS>
###### Abstract

This paper explores the state of the art on to methods to verify and validate
navigation algorithms for autonomous surface ships. We perform a systematic
mapping study to find research works published in the last 10 years proposing
new algorithms for autonomous navigation and collision avoidance and we have
extracted what verification and validation approaches have been applied on
these algorithms. We observe that most research works use simulations to
validate their algorithms. However, these simulations often involve just a few
scenarios designed manually. This raises the question if the algorithms have
been validated properly. To remedy this, we propose the use of a systematic
scenario-based testing approach to validate navigation algorithms extensively.

##  I Introduction

Maritime Autonomous Surface Ships (MASS) of the future will exhibit an
increasing range of self-sufficiency. Autonomous capabilities include
relieving the vessel operator from constant supervision by taking over certain
responsibilities of the vessels using partial or complete remote operation of
vessels, or partial or complete unsupervised navigation.

An important motivation for autonomous functions and increased intelligence in
ships is to improve safety, efficiency of operations and decrease the
environmental footprint. Despite the advances in technologies and constant
striving towards improved safety, accidents still happen. In 2017 alone, 3301
accidents were reported by the European Maritime Safety Agency and over 53% of
all reported accidents were collisions, contacts or grounding occurrences, all
due to navigational error [1]. The development of autonomous navigational
capabilities is seen as a possible solution to dramatically reduce the number
of accidents due to navigational error.

The use of autonomous navigational functions in vessels raises however the
question of what may happen if these autonomous functions have design defects.
This question is addressed by Valdez et al. [2] who present a hazard analysis
for the design phase of autonomous vessels. In this study, the authors
identify AI software failure as a hazard that can lead to many of the
identified accidents. Valdez proposes a number of safety controls to eliminate
or reduce the likelihood that software hazard appears but this study does not
address how to implement these safety controls. If we intend to use AI
software components in navigation algorithms, we must ensure that they work as
expected and we should be able to analyze and reveal whether these components
may contain faults.

Traditionally, navigation algorithms have been based on path planning and
optimization and have been designed manually. Programming is a notoriously
complex task and developing defect-free programs require the application of
correct by construction methods or an extensive verification and validation
effort.

An alternative to path planning and optimization algorithms is the use of
machine learning (ML), reinforcement learning (RL) and neural networks (NN).
Machine learning has shown staggering success in autonomous cars. Machine
learning is known to succeed and outperform traditional approaches specially
in vaguely defined problem domains, where it is difficult, if not impossible,
to create a full formal specification of the phenomenon under study. We
consider this to be the case for COLREGs-based navigation and we conjecture
that a ML-based navigation approach can outperform existing search-based and
optimization algorithms. Still, modern AI software may also contain faults
introduced during the learning process of a neural network. As an example,
Katz [3] has analyzed the deep neural network implementation of the next-
generation airborne collision avoidance system for unmanned aircraft (ACAS Xu)
and found that several logical requirements did not hold for the system as
well as some adversarial perturbations that could lead to erroneous collision
avoidance actions.

This paper explores the state of the art related to the methods used to verify
and validate surface ship navigation algorithms. For this, we have performed a
systematic mapping study to find research works published in the last 10 years
proposing new algorithms and we have extracted what verification and
validation approaches have been applied on these algorithms. We have observed
that most research works use simulations to validate their algorithms.
However, these simulations involve just a few scenarios, often designed
manually. Therefore, we propose the use of a systematic scenario-based testing
approach to validate navigation algorithms thoroughly.

We proceed as follows. The design of the mapping studied is presented in
Section 2, while its main results are presented in Section 3 and 4. Finally,
Section 5 describes the proposal for a method for validation of navigation
algorithm using systematic scenario-based testing.

##  II Study Design

We have adapted and applied the systematic mapping approach described in [4]
to the autonomous maritime domain. In this study, we first defined the
appropriate research questions, then conducted the search for the relevant
papers. Consequently, we filtered the obtained papers based on our predefined
inclusion and exclusion criteria. The result of our study, eventually, ended
in producing a systematic mapping.

###  II-A Research questions

The first step consists in defining the research question. In this study, we
define three main research questions. In order to structure the answer to the
main questions, we also defined a few sub-questions. Our research questions
(RQs) are as follows.

  * RQ1

What approaches for navigation or traffic avoidance in autonomous ships have
been presented in the research literature?

    1. (a)

When and where have they been published?

    2. (b)

What are the overall approaches?

    3. (c)

Do they involve single ship or a swarm of ships?

  * RQ2

What are the requirements for these approaches as presented in the research
literature?

    1. (a)

How the safety is defined?

    2. (b)

Are the requirements COLREGs compliant?

  * RQ3

How are these approaches verified and validated in the research literature?

###  II-B Search Strategy

The primary search is done in the _Web of Knowledge_ database, which includes
the core _Web of Science_ database as well as several regional databases. The
core Web of Science database consists of: _Science Citation Index Expanded
(1945-present)_ , _Social Sciences Citation Index (1956-present)_ , _Arts &
Humanities Citation Index (1975-present)_, _Conference Proceedings Citation
Index- Science (1990-present)_ , _Conference Proceedings Citation Index-
Social Science & Humanities (1990-present)_, and _Emerging Sources Citation
Index (2015-present)_. We opted for the papers published between 2010 and
2020. We defined the following criteria for our primary search.

(maritime $\lor$ marine $\lor$ ship $\lor$ vessel) $\wedge$ (autonomous
navigation $\lor$ autonomous traffic avoidance $\lor$ collision avoidance)
$\wedge$ (algorithm $\lor$ AI $\lor$ artificial intelligence $\lor$ machine
learning $\lor$ ML $\lor$ optimization $\lor$ optimisation) $\wedge$
(validation $\lor$ verification $\lor$ testing $\lor$ simulation $\lor$
quality $\lor$ safe $\lor$ safety)

This primary search resulted in the collection of 427 papers.

###  II-C Inclusion and Exclusion

At this step, we performed a screening process of the papers, considering only
relevant papers based on our inclusion and exclusion criteria. The adopted
inclusion criteria are: (1) Only peer-reviewed research papers published in a
journal or a conference proceeding; (2) Only papers related to the theme of
surface maritime vessel’s in their title or abstract or keywords; (3) Only
papers that mentioned machine learning or optimization algorithm in their
title or abstract or keywords. The exclusion criteria were: (1) Papers
mentioning “maritime vessel” in their abstract but that cannot be considered
as describing research on autonomy; (2) Papers are duplicates (3) Papers
containing keywords related to our study but discovered as false positives
(e.g. review papers, studies on underwater vessels). The full list of papers
was equally divided between the authors to apply the inclusion/exclusion
criteria and filter the relevant papers. The final list of papers after
applying the inclusion/exclusion criteria consisted of 132 papers.

The identified papers in the screening step were then randomly distributed
among four authors for the full reading step. As such, each paper was
processed by a second author, to avoid bias. The full list of proceed papers
could be found in the Appendix.

##  III Data extraction and classification

For the data extraction we followed the template presented in Table I.

Data Item | Value | RQ
---|---|---
General |  |
Study ID | Integer |
Paper Title | Title of the Paper |
Authors’ Name | List of Authors |
Year of Publication | Calendar Year | RQ1
Venue | Publication Venue Name | RQ1
Process |  |
Overall Approach | Algorithmic Approach | RQ1
Single or Swarm | Binary | RQ1
Safety | Safety Definition | RQ2
(Non) compliance with Regulations | Binary | RQ2
Verification & Validation | V&V Approach | RQ3
TABLE I: Data Extraction Form

We used the extracted data to answer our main research questions. Figure 1,
RQ1(a), presents the distribution of the studies between year 2010-2020. As it
can be seen from the graphs, the number of publications in the field
experienced a dramatic boost in year 2017. The majority of the studies were
published as a journal article, followed by conference papers and whole books,
87.5%, 9% and 3.5% respectively, see Figure 2, RQ1(a). This is to be expected
as the interest in autonomous vehicles have been piqued over the past few
years. As it could be observed from Figure 3, the majority of the papers opted
for optimization as their overall approach, RQ1(b). This indicates that the
use of AI is still at its infancy when it comes to autonomous navigation for
maritime surface vessels. Based on the data analysis results, 82% of the
studies involved only one single target ship, whereas the others, focus on a
swarm of ships, RQ1(c).

Figure 1: Publication Year Figure 2: Publication Type Figure 3: Overall
Approach

The majority of the articles defined safety based on the values of either Time
to Closest Point of Approach (TCPA) or Distance to Closest Point of Approach
(DCPA), 82%, RQ2(a). Only 48% of the papers chose to comply with CLOREGs in
their study design, RQ2(b), see Figure 4.

Figure 4: COLREG Compliance

The majority of the papers (86 out of 132) identified in this study used
simulation approaches to validate their results with a small (ranging from 1
to 12) number of scenarios. Three studies used either a real boat or a model
boat for the validation [76, 107, 25] and the rest did not use any
verification and validation approach, RQ3. The distribution of validation
methods is depicted in Figure 5

Figure 5: Verification & Validation Approaches

##  IV Current practice on the Verification and Validation of AI Navigation
Algorithms

The verification and validation of navigation algorithms is an important issue
since software failures has been identified as a hazard that can lead to many
accidents in vessels with autonomous functions. To avoid such hazard, Perera
proposes a 3-level approach to validate the behaviour of autonomous vessels,
[8]. Level 1 in Perera’s classification requires the use of a software
simulation for the motion of all vessels. A level 2 testing system would
require that the own ship is a full scale or model vessel that navigates in
restricted waters, while the other ships are simulated. In contrast, a level 3
system would require that all involved vessels navigate in open seas.

The mapping study show that most papers use software simulation to validate
the proposed results. In these simulations, the simulation starts with a given
scenario that describes the initial position and speeds for two or more
vessels. The scenario is then animated in the physics-based simulator and the
performance of the AI agents under test is evaluated. This corresponds to
Level 1 validation in Perera’s classification. However, we have observed that
most of these works simulate just some few scenarios and that these scenarios
are designed manually, often to represent standard situations such as a take
over or a crossing. Also, there is a considerable number of research articles
that do not contain any verification or validation of their proposed results.

Existing work in the verification and validation in the automotive domain
emphasizes the need to use a large number of specially designed scenarios in
order to be able to find some faults in autonomous functions. We consider that
the same criteria should apply to the maritime domain and that there is a need
for domain-specific methods for the systematic verification and validation of
autonomous functions in vessels. Therefore, we propose in the next section the
use of a systematic scenario-based testing approach to validate navigation
algorithms thoroughly.

##  V A Proposal for Navigation Algorithm Validation using Systematic
Scenario-Based Testing

The goal of scenario-based testing is to evaluate a large set scenarios to
find those where the AI agents do not perform as expected. In each scenario,
the position and the velocity vector of each ship may vary, as well as their
destination way-point. An example scenario with two vessels is depicted in
Fig. 6.

Figure 6: A possible scenario

We are interested to know if there are scenarios where the autonomous
validation components under study take the wrong decisions. These are
described as _challenging_ scenarios in the literature, and they lead to
undesirable outcomes such as near miss or a collision.

Testing a single scenario for an autonomous vehicle is computationally
expensive since it requires a physics-based simulation in addition to
executing the autonomous functions. This includes updating the motion of all
the vehicles involved in the scenario as well as simulating the environment
sensed by the autonomous functions. Since there is a limited testing budget
and we want to maximize the chances to find a defect, it is therefore
desirable to select the scenarios that are considered more challenging for the
autonomous function, [9].

Several authors have proposed methods to search for challenging scenarios
efficiently, [10, 11]. Abdessalem, Nejati, Vruand and Stifter have proposed a
method that uses neural networks as a surrogate model for the scenario fitness
functions and then genetic algorithms as a heuristic to search challenging
scenarios, [12]. This is presented as a two phase process. First a set of
simulations must be executed in order to create the surrogate models of the
fitness functions. Once these models have been created, the scenario search is
performed.

We have proposed a new approach for scenario-based testing that it is specific
to maritime surface vehicles and that avoid the need of training subrrogate
models. Our approach, presented in [13], is based on the use of a neural
network to discriminate and select scenarios that may be challenging for the
autonomous system being tested. The selected scenarios are simulated and
evaluated and their outcome is used to train the discriminating neural
network. Compared to other works such as [12], we combine the training of the
discriminator network and the scenario selection in one step, with the
intention to reduce the number of necessary simulations. The simulations are
evaluated by risk of collision and compliance to COLREGs.

To evaluate our approach, we have tested a collision avoidance algorithm based
on a neural network trained using reinforcement learning. The evaluation task
was to create 6000 simulation scenarios, each one depicting a different
initial situation. Our experimental results show that the proposed testing
method generates test suits composed mostly of challenging scenarios. This
allows us to validate quickly if the navigation algorithm under test can
operate safely while abiding the COLREGs.

##  VI Conclusions

This paper explores the state of the art on the methods to verify and validate
navigation algorithms for autonomous surface ships by carrying out a
systematic mapping study. The mapping study reveals that most research works
use simulations to validate their algorithms. Finally, we have proposed the
use of a systematic scenario-based testing approach to validate navigation
algorithms extensively.

## References

  * [1] EMSA, “Annual overview of marine casualties and incidents 2018.” European Maritime Safety Agency E.M.S. Agency, 2018.
  * [2] O. A. V. Banda, S. Kannos, F. Goerlandt, P. H. A. J. M. van Gelder, M. Bergström, and P. Kujala, “A systemic hazard analysis and management process for the concept design phase of an autonomous vessel,” Reliab. Eng. Syst. Saf., vol. 191, 2019.
  * [3] G. Katz, C. W. Barrett, D. L. Dill, K. Julian, and M. J. Kochenderfer, “Reluplex: An efficient SMT solver for verifying deep neural networks,” in Computer Aided Verification - 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I (R. Majumdar and V. Kuncak, eds.), vol. 10426 of Lecture Notes in Computer Science, pp. 97–117, Springer, 2017.
  * [4] K. Petersen, S. Vakkalanka, and L. Kuzniarz, “Guidelines for conducting systematic mapping studies in software engineering: An update,” Information and Software Technology, vol. 64, pp. 1–18, 2015.
  * [5] H. Shen, H. Hashimoto, A. Matsuda, Y. Taniguchi, D. Terada, and C. Guo, “Automatic collision avoidance of multiple ships based on deep q-learning,” Applied Ocean Research, vol. 86, pp. 268–288, 2019.
  * [6] J. Xin, S. Li, J. Sheng, Y. Zhang, and Y. Cui, “Application of improved particle swarm optimization for navigation of unmanned surface vehicles,” Sensors, vol. 19, no. 14, p. 3096, 2019.
  * [7] J. Han, Y. Cho, J. Kim, J. Kim, N.-s. Son, and S. Y. Kim, “Autonomous collision detection and avoidance for aragon usv: Development and field tests,” Journal of Field Robotics, 2020.
  * [8] L. P. Perera, “Deep Learning Toward Autonomous Ship Navigation and Possible COLREGs Failures,” Journal of Offshore Mechanics and Arctic Engineering, vol. 142, 12 2019.  031102\.
  * [9] D. Gagliardi, P. Tkachenko, and L. del Re, “Outcome oriented evaluation of autonomous driving functions,” in 57th IEEE Conference on Decision and Control, CDC 2018, Miami, FL, USA, December 17-19, 2018, pp. 6970–6975, IEEE, 2018.
  * [10] R. B. Abdessalem, S. Nejati, L. C. Briand, and T. Stifter, “Testing vision-based control systems using learnable evolutionary algorithms,” in Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018 (M. Chaudron, I. Crnkovic, M. Chechik, and M. Harman, eds.), pp. 1016–1026, ACM, 2018.
  * [11] G. E. Mullins, P. G. Stankiewicz, and S. K. Gupta, “Automated generation of diverse and challenging scenarios for test and evaluation of autonomous vehicles,” in 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29 - June 3, 2017, pp. 1443–1450, IEEE, 2017.
  * [12] R. B. Abdessalem, S. Nejati, L. C. Briand, and T. Stifter, “Testing advanced driver assistance systems using multi-objective search and neural networks,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016 (D. Lo, S. Apel, and S. Khurshid, eds.), pp. 63–74, ACM, 2016.
  * [13] I. Porres, S. Azimi, and J. Lilius, “Scenario-based testing of a ship collision avoidance system,” in 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), IEEE, 2020.

## Articles included in the Mapping Study

## References

  * [1] Karl Gunnar Aarsæther and Torgeir Moan.  Adding the human element to ship manoeuvring simulations.  The Journal of Navigation, 63(4):695, 2010.
  * [2] Mohamed Abdelaal and Axel Hahn.  Nmpc-based trajectory tracking and collision avoidance of unmanned surface vessels with rule-based colregs confinement.  In 2016 IEEE Conference on Systems, Process and Control (ICSPC), pages 23–28. IEEE, 2016.
  * [3] Jin-Hyeong Ahn, Key-Pyo Rhee, and Young-Jun You.  A study on the collision avoidance of a ship using neural networks and fuzzy logic.  Applied Ocean Research, 37:162–173, 2012.
  * [4] Azzeddine Bakdi, Ingrid Kristine Glad, Erik Vanem, and Øystein Engelhardtsen.  Ais-based multiple vessel collision and grounding risk identification based on adaptive safety domain.  Journal of Marine Science and Engineering, 8(1):5, 2020.
  * [5] Marco Bibuli, Yogang Singh, Sanjay Sharma, Robert Sutton, Daniel Hatton, and Asiya Khan.  A two layered optimal approach towards cooperative motion planning of unmanned surface vehicles in a constrained maritime environment.  IFAC-PapersOnLine, 51(29):378–383, 2018.
  * [6] Morten Breivik et al.  Mpc-based mid-level collision avoidance for asvs using nonlinear programming.  In 2017 IEEE Conference on Control Technology and Applications (CCTA), pages 766–772. IEEE, 2017.
  * [7] Mauro Candeloro, Anastasios M Lekkas, and Asgeir J Sørensen.  A voronoi-diagram-based dynamic path-planning system for underactuated marine vessels.  Control Engineering Practice, 61:41–54, 2017.
  * [8] Wu Chao, Ma Feng, Wu Qing, and Wang Shuwu.  A situation awareness approach for usv based on artificial potential fields.  In 2017 4th International Conference on Transportation Information and Safety (ICTIS), pages 232–235. IEEE, 2017.
  * [9] Chen Chen, Xian-Qiao Chen, Feng Ma, Xiao-Jun Zeng, and Jin Wang.  A knowledge-free path planning approach for smart ships based on reinforcement learning.  Ocean Engineering, 189:106299, 2019.
  * [10] Siyu Guo, Xiuguo Zhang, Yisong Zheng, and Yiquan Du.  An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning.  Sensors, 20(2), JAN 2020.
  * [11] Linying Chen, Hans Hopman, and Rudy R Negenborn.  Distributed model predictive control for vessel train formations of cooperative multi-vessel systems.  Transportation Research Part C: Emerging Technologies, 92:101–118, 2018.
  * [12] PF Chen, PHAJM van Gelder, and JM Mou.  Integration of elliptical ship domains and velocity obstacles for ship collision candidate detection.  TransNav, International Journal on Marine Navigation and Safety od Sea Transportation, 13(4), 2019.
  * [13] Jinwoo Choi, Jeonghong Park, Jongdae Jung, Yoongeon Lee, and Hyun-Taek Choi.  Development of an autonomous surface vehicle and performance evaluation of autonomous navigation technologies.  International Journal of Control, Automation and Systems, 18(3):535–545, 2020.
  * [14] Shejun Deng, Yuyi Zhong, Zhijin Wu, Min Wang, Lvkai Zhu, and Hongru Yu.  Research on waterborne collision avoidance and early warning algorithm based on location data.  Advances in Mechanical Engineering, 12(2):1687814020906079, 2020\.
  * [15] Zaopeng Dong, Tao Bao, Mao Zheng, Xin Yang, Lifei Song, and Yunsheng Mao.  Heading control of unmanned marine vehicles based on an improved robust adaptive fuzzy neural network control algorithm.  IEEE Access, 7:9704–9713, 2019.
  * [16] Lei Du, Floris Goerlandt, Osiris A Valdez Banda, Yamin Huang, Yuanqiao Wen, and Pentti Kujala.  Improving stand-on ship’s situational awareness by estimating the intention of the give-way ship.  Ocean Engineering, 201:107110, 2020.
  * [17] Armagan Elibol, Nuno Gracias, and Rafael Garcia.  Augmented state–extended kalman filter combined framework for topology estimation in large-area underwater mapping.  Journal of Field Robotics, 27(5):656–674, 2010.
  * [18] Bjørn-Olav H Eriksen, Glenn Bitar, Morten Breivik, and Anastasios M Lekkas.  Hybrid collision avoidance for asvs compliant with colregs rules 8 and 13–17.  Frontiers in Robotics and AI, 7:11, 2020.
  * [19] Bjørn-Olav H Eriksen, Morten Breivik, Erik F Wilthil, Andreas L Flåten, and Edmund F Brekke.  The branching-course model predictive control algorithm for maritime collision avoidance.  Journal of Field Robotics, 36(7):1222–1249, 2019.
  * [20] R Fişkin, H Kişi, and E Nasibov.  A research on techniques, models and methods proposed for ship collision avoidance path planning problem.  International Journal Maritime Engineering, 160(A2):187–206, 2018\.
  * [21] R Fiskin, E Nasibov, and MO Yardimci.  Deterministic-based ship anti-collision route optimization with web-based application.  International Journal of Maritime Engineering, 161:A345–A356, 2019\.
  * [22] Ming-yu Fu, Sha-sha Wang, and Yuan-hui Wang.  Multi-behavior fusion based potential field method for path planning of unmanned surface vessel.  China Ocean Engineering, 33(5):583–592, 2019.
  * [23] Xiongfei Geng, Yongcai Wang, Ping Wang, and Baochen Zhang.  Motion plan of maritime autonomous surface ships by dynamic programming for collision avoidance and speed optimization.  Sensors, 19(2):434, 2019.
  * [24] Siyu Guo, Xiuguo Zhang, Yisong Zheng, and Yiquan Du.  An autonomous path planning model for unmanned ships based on deep reinforcement learning.  Sensors, 20(2):426, 2020.
  * [25] Jungwook Han, Yonghoon Cho, Jonghwi Kim, Jinwhan Kim, Nam-sun Son, and Sun Young Kim.  Autonomous collision detection and avoidance for aragon usv: Development and field tests.  Journal of Field Robotics, 2020.
  * [26] Qilong Han, Xiao Yang, Hongtao Song, Shanshan Sui, Hui Zhang, and Zaiqiang Yang.  Whale optimization algorithm for ship path optimization in large-scale complex marine environment.  IEEE Access, 8:57168–57179, 2020.
  * [27] Wei He, Shuo Xie, Xinglong Liu, Tao Lu, Tianjiao Luo, Miguel Angel Sotelo, and Zhixiong Li.  A novel image recognition algorithm of target identification for unmanned surface vehicles based on deep learning.  Journal of Intelligent & Fuzzy Systems, 37(4):4437–4447, 2019\.
  * [28] Ramdane Hedjar and Messaoud Bounkhel.  An automatic collision avoidance algorithm for multiple marine surface vehicles.  International Journal of Applied Mathematics and Computer Science, 29(4):759–768, 2019.
  * [29] MA Hinostroza and C Guedes Soares.  Collision avoidance, guidance and control system for autonomous surface vehicles in complex navigation conditions.  Progress in Maritime Technology and Engineering, pp. Taylor & Francis Group, London, UK, pages 121–132, 2018.
  * [30] Liang Hu, Wasif Naeem, Eshan Rajabally, Graham Watson, Terry Mills, Zakirul Bhuiyan, Craig Raeburn, Ivor Salter, and Claire Pekcan.  A multiobjective optimization approach for colregs-compliant path planning of autonomous surface vehicles verified on networked bridge simulators.  IEEE Transactions on Intelligent Transportation Systems, 21(3):1167–1179, 2019.
  * [31] Liang Hu, Wasif Naeem, Eshan Rajabally, Graham Watson, Terry Mills, Zakirul Bhuiyan, and Ivor Salter.  Colregs-compliant path planning for autonomous surface vehicles: A multiobjective optimization approach.  IFAC-PapersOnLine, 50(1):13662–13667, 2017.
  * [32] Qing Hu, Yi Jiang, Jingbo Zhang, Xiaowen Sun, and Shufang Zhang.  Development of an automatic identification system autonomous positioning system.  Sensors, 15(11):28574–28591, 2015.
  * [33] Yamin Huang, Linying Chen, Pengfei Chen, Rudy R Negenborn, and PHAJM van Gelder.  Ship collision avoidance methods: State-of-the-art.  Safety science, 121:451–473, 2020.
  * [34] Yamin Huang, Linying Chen, and PHAJM van Gelder.  Generalized velocity obstacle algorithm for preventing ship collisions at sea.  Ocean Engineering, 173:142–156, 2019.
  * [35] Yamin Huang, PHAJM van Gelder, and Yuanqiao Wen.  Velocity obstacle algorithms for collision prevention at sea.  Ocean Engineering, 151:308–321, 2018.
  * [36] Timur İnan and Ahmet Fevzi Baba.  Particle swarm optimization-based collision avoidance.  Turkish Journal of Electrical Engineering and Computer Science, 27(3):2137–2155, 2019.
  * [37] Min-Gi Jeong, Eun-Bang Lee, and Moonjin Lee.  An adaptive route plan technique with risk contour for autonomous navigation of surface vehicles.  In OCEANS 2018 MTS/IEEE Charleston, pages 1–4. IEEE, 2018.
  * [38] Min-Gi Jeong, Eun-Bang Lee, Moonjin Lee, and Jung-Yeul Jung.  Multi-criteria route planning with risk contour map for smart navigation.  Ocean Engineering, 172:72–85, 2019.
  * [39] Yu-Tao Kang, Wei-Jiong Chen, Da-Qi Zhu, Jin-Hui Wang, and Qi-Miao Xie.  Collision avoidance path planning for ships by particle swarm optimization.  Journal of Marine Science and Technology, 26(6):777–786, 2018.
  * [40] Joanna Karbowska-Chilinska, Jolanta Koszelew, Krzysztof Ostrowski, Piotr Kuczynski, Eric Kulbiej, and Piotr Wolejsza.  Beam search algorithm for ship anti-collision trajectory planning.  Sensors, 19(24):5338, 2019.
  * [41] Heesu Kim, Sang-Hyun Kim, Maro Jeon, JaeHak Kim, Soonseok Song, and Kwang-Jun Paik.  A study on path optimization method of an unmanned surface vehicle under environmental loads using genetic algorithm.  Ocean Engineering, 142:616–624, 2017.
  * [42] Łukasz Kuczkowski and Roman Śmierzchalski.  Comparison of single and multi-population evolutionary algorithm for path planning in navigation situation.  In Solid State Phenomena, volume 210, pages 166–177. Trans Tech Publ, 2014.
  * [43] Lukasz Kuczkowski and Roman Smierzchalski.  Termination functions for evolutionary path planning algorithm.  In 2014 19th International Conference on Methods and Models in Automation and Robotics (MMAR), pages 636–640. IEEE, 2014.
  * [44] Yoshiaki Kuwata, Michael T Wolf, Dimitri Zarzhitsky, and Terrance L Huntsberger.  Safe maritime navigation with colregs using velocity obstacles.  In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4728–4734. IEEE, 2011.
  * [45] A Lazarowska.  An efficient graph theory-based algorithm for ship trajectory planning.  INTERNATIONAL JOURNAL OF MARITIME ENGINEERING, 161:155–161, 2019\.
  * [46] Agnieszka Lazarowska.  Swarm intelligence approach to safe ship control.  Polish Maritime Research, 22(4):34–40, 2015.
  * [47] Agnieszka Lazarowska.  Research on algorithms for autonomous navigation of ships.  WMU Journal of Maritime Affairs, 18(2):341–358, 2019.
  * [48] Man-Chun Lee, Chung-Yuan Nieh, Hsin-Chuan Kuo, and Juan-Chen Huang.  An automatic collision avoidance and route generating algorithm for ships based on field model.  Journal of Marine Science and Technology, 27(2):101–113, 2019.
  * [49] Jinxin Li, Hongbo Wang, Wei Zhao, and Yuanyuan Xue.  Ship’s trajectory planning based on improved multiobjective algorithm for collision avoidance.  Journal of Advanced Transportation, 2019, 2019.
  * [50] Shijie Li, Jialun Liu, and Rudy R Negenborn.  Distributed coordination for collision avoidance of multiple ships considering ship maneuverability.  Ocean Engineering, 181:212–226, 2019.
  * [51] Shijie Li, Jialun Liu, Rudy R Negenborn, and Feng Ma.  Optimizing the joint collision avoidance operations of multiple ships from an overall perspective.  Ocean Engineering, 191:106511, 2019.
  * [52] Weifeng Li and Wenyao Ma.  Simulation on vessel intelligent collision avoidance based on artificial fish swarm algorithm.  Polish Maritime Research, 23(s1):138–143, 2016.
  * [53] J Lisowski.  Multi-criteria optimization of multi-step matrix game in collision avoidance of ships.  TransNav: International Journal on Marine Navigation and Safety of Sea Transportation, 13, 2019.
  * [54] Józef Lisowski.  Computational intelligence methods of a safe ship control.  Procedia computer science, 35:634–643, 2014.
  * [55] Deli Liu, Dong Xu, Nan Wang, Ziying Zhang, and Pingpeng Tang.  Dynamic replanning algorithm of local trajectory for unmanned surface vehicle.  In 2016 35th Chinese Control Conference (CCC), pages 7120–7125. IEEE, 2016.
  * [56] Hongdan Liu, Sheng Liu, and Lanyong Zhang.  Ship collision avoidance path planning strategy based on quantum bacterial foraging algorithm.  In 2015 2nd International Conference on Electrical, Computer Engineering and Electronics. Atlantis Press, 2015.
  * [57] Hongdan Liu, Rong Sun, and Qi Liu.  The tactics of ship collision avoidance based on quantum-behaved wolf pack algorithm.  Concurrency and Computation: Practice and Experience, 32(6):e5196, 2020.
  * [58] Shuchen Liu, Sylvain Roy, Eloy Pairet-Garcia, Jan-Jöran Gehrt, Friederike Siemer, Christof Büskens, Dirk Abel, and René Zweigel.  Case study: Networked control for optimal maneuvering of autonomous vessels.  IFAC-PapersOnLine, 52(8):440–445, 2019.
  * [59] Xinyu Liu, Yun Li, Jing Zhang, Jian Zheng, and Chunxi Yang.  Self-adaptive dynamic obstacle avoidance and path planning for usv under complex maritime environment.  IEEE Access, 7:114945–114954, 2019.
  * [60] Yuanchang Liu and Richard Bucknall.  Efficient multi-task allocation and path planning for unmanned surface vehicle in support of ocean operations.  Neurocomputing, 275:1550–1566, 2018.
  * [61] Yuanchang Liu, Richard Bucknall, and Xinyu Zhang.  The fast marching method based intelligent navigation of an unmanned surface vehicle.  Ocean Engineering, 142:363–376, 2017.
  * [62] Yuanchang Liu, Rui Song, Richard Bucknall, and Xinyu Zhang.  Intelligent multi-task allocation and planning for multiple unmanned surface vehicles (usvs) using self-organising maps and fast marching method.  Information Sciences, 496:180–197, 2019.
  * [63] Hongguang Lyu and Yong Yin.  Fast path planning for autonomous ships in restricted waters.  Applied Sciences, 8(12):2592, 2018.
  * [64] Hongguang Lyu and Yong Yin.  Colregs-constrained real-time path planning for autonomous ships using modified artificial potential fields.  The Journal of Navigation, 72(3):588–608, 2019.
  * [65] LY Ma, Wei Xie, and HB Huang.  Convolutional neural network based obstacle detection for unmanned surface vehicle.  Mathematical biosciences and engineering: MBE, 17(1):845–861, 2019\.
  * [66] Yong Ma, Mengqi Hu, and Xinping Yan.  Multi-objective path planning for unmanned surface vehicle with currents effects.  ISA transactions, 75:137–156, 2018.
  * [67] Mostefa Mohamed-Seghir.  Methods based on fuzzy sets to solve problems of safe ship control.  In Novel Algorithms and Techniques in Telecommunications and Networking, pages 373–377. Springer, 2010.
  * [68] Wasif Naeem, Sable C Henrique, and Liang Hu.  A reactive colregs-compliant navigation strategy for autonomous maritime navigation.  IFAC-PapersOnLine, 49(23):207–213, 2016.
  * [69] Wasif Naeem, George W Irwin, and Aolei Yang.  Colregs-based collision avoidance strategies for unmanned surface vehicles.  Mechatronics, 22(6):669–678, 2012.
  * [70] Hanlin Niu, Al Savvaris, and Antonios Tsourdos.  Usv geometric collision avoidance algorithm for multiple marine vehicles.  In OCEANS 2017-Anchorage, pages 1–10. IEEE, 2017.
  * [71] Bartosz Ożoga and Jakub Montewka.  Towards a decision support system for maritime navigation on heavily trafficked basins.  Ocean Engineering, 159:88–97, 2018.
  * [72] Madhusmita Panda, Bikramaditya Das, Bidyadhar Subudhi, and Bibhuti Bhusan Pati.  A comprehensive review of path planning algorithms for autonomous underwater vehicles.  International Journal of Automation and Computing, pages 1–32, 2020\.
  * [73] Giulia Pedrielli, Yifan Xing, Jia Hao Peh, Kim Wee Koh, and Szu Hui Ng.  A real time simulation optimization framework for vessel collision avoidance and the case of singapore strait.  IEEE Transactions on Intelligent Transportation Systems, 21(3):1204–1215, 2019.
  * [74] Zihe Qin, Zhuang Lin, Dongmei Yang, and Ping Li.  A task-based hierarchical control strategy for autonomous motion of an unmanned surface vehicle swarm.  Applied Ocean Research, 65:251–261, 2017.
  * [75] Andrzej Rak and Witold Gierusz.  Reinforcement learning in discrete and continuous domains applied to ship trajectory generation.  Polish Maritime Research, 19(Special):31–36, 2012.
  * [76] Haiqing Shen, Hirotada Hashimoto, Akihiko Matsuda, Yuuki Taniguchi, Daisuke Terada, and Chen Guo.  Automatic collision avoidance of multiple ships based on deep q-learning.  Applied Ocean Research, 86:268–288, 2019.
  * [77] Yogang Singh, Sanjay Sharma, Robert Sutton, Daniel Hatton, and Asiya Khan.  A constrained a* approach towards optimal path planning for an unmanned surface vehicle in a maritime environment containing dynamic obstacles and ocean currents.  Ocean Engineering, 169:187–201, 2018.
  * [78] Yogang Singh, Sanjay Sharma, Robert Sutton, Daniel Hatton, and Asiya Khan.  Feasibility study of a constrained dijkstra approach for optimal path planning of an unmanned surface vehicle in a dynamic maritime environment.  In 2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pages 117–122. IEEE, 2018.
  * [79] A Lifei Song, B Yiran Su, C Zaopeng Dong, D Wei Shen, E Zuquan Xiang, and F Puxiu Mao.  A two-level dynamic obstacle avoidance algorithm for unmanned surface vehicles.  Ocean Engineering, 170:351–360, 2018.
  * [80] Lifei Song, Zhuo Chen, Zaopeng Dong, Zuquan Xiang, Yunsheng Mao, Yiran Su, and Kai Hu.  Collision avoidance planning for unmanned surface vehicle based on eccentric expansion.  International Journal of Advanced Robotic Systems, 16(3):1729881419851945, 2019.
  * [81] Rui Song, Yuanchang Liu, and Richard Bucknall.  A multi-layered fast marching method for unmanned surface vehicle path planning in a time-variant maritime environment.  Ocean Engineering, 129:301–317, 2017.
  * [82] Xiaojie Sun, Guofeng Wang, Yunsheng Fan, Dongdong Mu, and Bingbing Qiu.  An automatic navigation system for unmanned surface vehicles in realistic sea environments.  Applied Sciences, 8(2):193, 2018.
  * [83] Xiaojie Sun, Guofeng Wang, Yunsheng Fan, Dongdong Mu, and Bingbing Qiu.  Collision avoidance of podded propulsion unmanned surface vehicle with colregs compliance and its modeling and identification.  IEEE Access, 6:55473–55491, 2018.
  * [84] Rafal Szlapczynski.  Evolutionary sets of safe ship trajectories within traffic separation schemes.  The Journal of Navigation, 66(1):65–81, 2013.
  * [85] Rafal Szlapczynski.  Evolutionary planning of safe ship tracks in restricted visibility.  The Journal of Navigation, 68(1):39–51, 2015.
  * [86] Rafał Szłapczyński and Hossein Ghaemi.  Framework of an evolutionary multi-objective optimisation method for planning a safe trajectory for a marine autonomous surface ship.  Polish Maritime Research, 26(4):69–79, 2019.
  * [87] Rafał Szłapczyński and Joanna Szłapczyńska.  Evolutionary sets of safe ship trajectories: Problem dedicated operators.  In International Conference on Computational Collective Intelligence, pages 221–230. Springer, 2011.
  * [88] CheeKuang Tam and Richard Bucknall.  Path-planning algorithm for ships in close-range encounters.  Journal of marine science and technology, 15(4):395–407, 2010.
  * [89] Guoge Tan, Jin Zou, Jiayuan Zhuang, Lei Wan, Hanbing Sun, and Zhiyuan Sun.  Fast marching square method based intelligent navigation of the unmanned surface vehicle swarm in restricted waters.  Applied Ocean Research, 95:102018, 2020.
  * [90] Ming-Cheng Tsou, Sheng-Long Kao, and Chien-Min Su.  Decision support from genetic algorithms for ship collision avoidance route planning and alerts.  The Journal of Navigation, 63(1):167, 2010.
  * [91] Sebastián Aldo Villar et al.  Navigation system for macábot an autonomous surface vehicles using gps aided strapdown inertial navigation system.  IEEE Latin America Transactions, 17(06):1009–1019, 2019.
  * [92] C Wang, YS Mao, KJ Du, BQ Hu, and LF Song.  Simulation on local obstacle avoidance algorithm for unmanned surface vehicle.  International Journal of Simulation Modelling, 15(3):460–472, 2016\.
  * [93] Chengbo Wang, Xinyu Zhang, Longze Cong, Junjie Li, and Jiawei Zhang.  Research on intelligent collision avoidance decision-making of unmanned ship in unknown environments.  Evolving Systems, 10(4):649–658, 2019.
  * [94] Chengbo Wang, Xinyu Zhang, Ruijie Li, and Peifang Dong.  Path planning of maritime autonomous surface ships in unknown environment with reinforcement learning.  In International Conference on Cognitive Systems and Signal Processing, pages 127–137. Springer, 2018.
  * [95] Hongjian Wang and Xicheng Ban.  Research on autonomous collision avoidance method of unmanned surface vessel in the circumstance of moving obstacles.  In 2018 37th Chinese Control Conference (CCC), pages 501–506. IEEE, 2018.
  * [96] Ning Wang, Yuncheng Gao, Zhongjiu Zheng, Hong Zhao, and Jianchuan Yin.  A hybrid path-planning scheme for an unmanned surface vehicle.  In 2018 Eighth International Conference on Information Science and Technology (ICIST), pages 231–236. IEEE, 2018.
  * [97] Ning Wang, Xiaozhao Jin, and Meng Joo Er.  A multilayer path planner for a usv under complex marine environments.  Ocean Engineering, 184:1–10, 2019.
  * [98] Ning Wang, Yue Tan, and Shao-Man Liu.  Ship domain identification using fast and accurate online self-organizing parsimonious fuzzy neural networks.  In Proceedings of the 30th Chinese Control Conference, pages 5271–5276. IEEE, 2011.
  * [99] Zhaokun Wei, Kang Zhao, and Ming Wei.  Decision-making in ship collision avoidance based on cat-swarm biological algorithm.  In 2015 International Conference on Computational Science and Engineering. Atlantis Press, 2015.
  * [100] Martin S Wiig, Kristin Y Pettersen, and Thomas R Krogstad.  A reactive collision avoidance algorithm for vehicles with underactuated dynamics.  In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pages 1452–1459. IEEE, 2017.
  * [101] Martin Syre Wiig, Kristin Ytterstad Pettersen, and Thomas Røbekk Krogstad.  Collision avoidance for underactuated marine vehicles using the constant avoidance angle algorithm.  IEEE Transactions on Control Systems Technology, 28(3):951–966, 2019.
  * [102] Kyle Woerner and Michael Benjamin.  Safety and efficiency analysis of autonomous collision avoidance using multi-objective optimization with interval programming.  Naval Engineers Journal, 126(4):163–168, 2014.
  * [103] Joohyun Woo and Nakwan Kim.  Collision avoidance for an unmanned surface vehicle using deep reinforcement learning.  Ocean Engineering, 199:107001, 2020.
  * [104] R Glenn Wright.  Intelligent autonomous ship navigation using multi-sensor modalities.  TransNav: International Journal on Marine Navigation and Safety of Sea Transportation, 13(3), 2019.
  * [105] Shuo Xie, Xiumin Chu, Mao Zheng, and Chenguang Liu.  Ship predictive collision avoidance method based on an improved beetle antennae search algorithm.  Ocean Engineering, 192:106542, 2019.
  * [106] Shuo Xie, Vittorio Garofano, Xiumin Chu, and Rudy R Negenborn.  Model predictive ship collision avoidance based on q-learning beetle swarm antenna search and neural networks.  Ocean Engineering, 193:106609, 2019.
  * [107] Junfeng Xin, Shixin Li, Jinlu Sheng, Yongbo Zhang, and Ying Cui.  Application of improved particle swarm optimization for navigation of unmanned surface vehicles.  Sensors, 19(14):3096, 2019.
  * [108] Junfeng Xin, Jiabao Zhong, Fengru Yang, Ying Cui, and Jinlu Sheng.  An improved genetic algorithm for path-planning of unmanned surface vehicle.  Sensors, 19(11):2640, 2019.
  * [109] Chengke Xiong, Danfeng Chen, Di Lu, Zheng Zeng, and Lian Lian.  Path planning of multiple autonomous marine vehicles for adaptive sampling using voronoi-based ant colony optimization.  Robotics and Autonomous Systems, 115:90–103, 2019.
  * [110] Haitong Xu, Hao Rong, and C Guedes Soares.  Use of ais data for guidance and control of path-following autonomous vessels.  Ocean Engineering, 194:106635, 2019.
  * [111] Qingyang Xu.  Collision avoidance strategy optimization based on danger immune algorithm.  Computers & Industrial Engineering, 76:268–279, 2014.
  * [112] Yan Zhuo Xue, Yi Wei, and Yue Qiao.  The research on ship intelligence navigation in confined waters.  In Advanced Materials Research, volume 442, pages 398–401. Trans Tech Publ, 2012.
  * [113] Yanzhuo Xue, D Clelland, BS Lee, and Duanfeng Han.  Automatic simulation of ship navigation.  Ocean Engineering, 38(17-18):2290–2305, 2011.
  * [114] Rongjie Yan, Xiangtong Yao, Junjie Yang, and Kai Huang.  Formal collision avoidance analysis for rigorous building of autonomous marine vehicles.  In National Conference on Embedded System Technology, pages 118–127. Springer, 2017.
  * [115] Hui Yang, Jie Qi, Yongchun Miao, Haixin Sun, and Jianghui Li.  A new robot navigation algorithm based on a double-layer ant algorithm and trajectory optimization.  IEEE Transactions on Industrial Electronics, 66(11):8557–8566, 2018\.
  * [116] Rongwu Yang, Jinsong Xu, Xin Wang, and Quan Zhou.  Parallel trajectory planning for shipborne autonomous collision avoidance system.  Applied Ocean Research, 91:101875, 2019.
  * [117] Tingting Yang, Chengzhuo Han, Meng Qin, and Chuan Huang.  Learning-aided intelligent cooperative collision avoidance mechanism in dynamic vessel networks.  IEEE Transactions on Cognitive Communications and Networking, 6(1):74–82, 2019.
  * [118] Raphael Zaccone and Michele Martelli.  A collision avoidance algorithm for ship guidance applications.  Journal of Marine Engineering & Technology, 19(sup1):62–75, 2020\.
  * [119] Raphael Zaccone, Michele Martelli, and Massimo Figari.  A colreg-compliant ship collision avoidance algorithm.  In 2019 18th European Control Conference (ECC), pages 2530–2535. IEEE, 2019.
  * [120] Zheng Zeng, Karl Sammut, Lian Lian, Andrew Lammas, Fangpo He, and Youhong Tang.  Rendezvous path planning for multiple autonomous marine vehicles.  IEEE Journal of Oceanic Engineering, 43(3):640–664, 2017.
  * [121] J Zhang, Q Hu, and B Liao.  Ship collision avoidance decision model and simulation based on collision circle.  TransNav: International Journal on Marine Navigation and Safety of Sea Transportation, 13(2), 2019.
  * [122] RL Zhang and M Furusho.  Conversion timing of seafarer’s decision-making for unmanned ship navigation.  TransNav: International Journal on Marine Navigation and Safety of Sea Transportation, 11(3), 2017.
  * [123] Xinyu Zhang, Chengbo Wang, Yuanchang Liu, and Xiang Chen.  Decision-making for the autonomous navigation of maritime autonomous surface ships based on scene division and deep reinforcement learning.  Sensors, 19(18):4055, 2019.
  * [124] Luman Zhao and Myung-Il Roh.  Colregs-compliant multiship collision avoidance based on deep reinforcement learning.  Ocean Engineering, 191:106436, 2019.
  * [125] Luman Zhao, Myung-Il Roh, and Sung-Jun Lee.  Control method for path following and collision avoidance of autonomous ship based on deep reinforcement learning.  Journal of Marine Science and Technology, 27(4):293–310, 2019.
  * [126] Yujiao Zhao, Xin Qi, Atilla Incecik, Yong Ma, and Zhixiong Li.  Broken lines path following algorithm for a water-jet propulsion usv with disturbance uncertainties.  Ocean Engineering, 201:107118, 2020.
  * [127] Yuxin Zhao, Wang Li, and Peng Shi.  A real-time collision avoidance learning system for unmanned surface vessels.  Neurocomputing, 182:255–266, 2016.
  * [128] Huarong Zheng, Rudy R Negenborn, and Gabriël Lodewijks.  Fast admm for distributed model predictive control of cooperative waterborne agvs.  IEEE Transactions on Control Systems Technology, 25(4):1406–1413, 2016.
  * [129] Kai Zheng, Yabo Chen, Yi Jiang, and Shuanghu Qiao.  A svm based ship collision risk assessment algorithm.  Ocean Engineering, 202:107062, 2020.
  * [130] Xinyuan Zhou, Peng Wu, Haifeng Zhang, Weihong Guo, and Yuanchang Liu.  Learn to navigate: cooperative path planning for unmanned surface vehicles using deep reinforcement learning.  IEEE Access, 7:165262–165278, 2019.
  * [131] Jiayuan Zhuang, Lei Zhang, Zihe Qin, Hanbing Sun, Bo Wang, and Jian Cao.  Motion control and collision avoidance algorithms for unmanned surface vehicle swarm in practical maritime environment.  Polish Maritime Research, 26(1):107–116, 2019.
  * [132] S Zinchenko, P Nosov, V Mateichuk, P Mamenko, I Popovych, and O Grosheva.  Automatic collision avoidance system with many targets, including maneuvering ones.  Bulletin of The University of Karaganda-Physics, 4(96):69–79, 2019.


# Indescribable Multi-modal Spatial Evaluator

Lingke Kong
Manteia Tech
<EMAIL_ADDRESS>X. Sharon Qi
University of California, Los Angeles
<EMAIL_ADDRESS>Qijin Shen
Fuzhou University
<EMAIL_ADDRESS>Jiacheng Wang
Xiamen University
<EMAIL_ADDRESS>Jingyi Zhang
Xiamen University
<EMAIL_ADDRESS>Yanle Hu111Corresponding author.
Mayo Clinic Arizona
<EMAIL_ADDRESS>Qichao Zhou111Corresponding author.
Manteia Tech
<EMAIL_ADDRESS>
###### Abstract

Multi-modal image registration spatially aligns two images with different
distributions. One of its major challenges is that images acquired from
different imaging machines have different imaging distributions, making it
difficult to focus only on the spatial aspect of the images and ignore
differences in distributions. In this study, we developed a self-supervised
approach, Indescribable Multi-model Spatial Evaluator (IMSE), to address
multi-modal image registration. IMSE creates an accurate multi-modal spatial
evaluator to measure spatial differences between two images, and then
optimizes registration by minimizing the error predicted of the evaluator. To
optimize IMSE performance, we also proposed a new style enhancement method
called Shuffle Remap which randomizes the image distribution into multiple
segments, and then randomly disorders and remaps these segments, so that the
distribution of the original image is changed. Shuffle Remap can help IMSE to
predict the difference in spatial location from unseen target distributions.
Our results show that IMSE outperformed the existing methods for registration
using T1-T2 and CT-MRI datasets. IMSE also can be easily integrated into the
traditional registration process, and can provide a convenient way to evaluate
and visualize registration results. IMSE also has the potential to be used as
a new paradigm for image-to-image translation. Our code is available at
https://github.com/Kid-Liet/IMSE.

##  1 Introduction

The purpose of multi-modal image registration is to align two images with
different distributions (Moving ($M$) and Target ($T$) images) by warping the
space through the deformation field $\phi$. A major challenge in multi-modal
image registration is that images from different modalities may differ in
multiple aspects given the fact that images are acquired using different
imaging machines, or different acquisition parameters. Due to dramatically
different reconstruction and acquisition methods, there is no simple one-to-
one mapping between different imaging modalities. From the perspective of
measuring similarity, mainstream unsupervised multi-modal registration methods
can be divided into two categories: similarity operator based registration and
image-to-image translation based registration.

Figure 1: The GAN based methods can only ensure that the distribution of the
$X$ domain is mapped that of the $Y$ domain. Ideally, we want to achieve
instance registration in which moving and target images are one-to-one
corresponded.

Similarity operator based registration uses multi-modal similarity operators
as loss functions for registration, for example, normalized cross-correlation
(NCC) [32, 31, 22, 15], mutual information (MI) [27, 23, 5, 35], and modality-
independent neighborhood descriptor (MIND) [13, 8, 39, 3]. Similarity
operators are based on a prior mathematical knowledge. These are carefully
designed and improved over time. They can be applied to both traditional
registration process (Eq. 1) and neural network registration (Eq. 2):

$\displaystyle\hat{\phi}=\mathop{\arg\min}_{\phi}\mathcal{L}_{sim}\left(M\left(\phi\right),T\right).$
(1)

Or

$\displaystyle\hat{\theta}=\mathop{\arg\min}_{\theta}\left[\mathbb{E}_{\left(M,T\right)}\left[\mathcal{L}_{sim}\left(M,T,g_{\theta}\left(M,T\right)\right)\right]\right].$
(2)

Similarity operators have several limitations. 1) It is unlikely to design
similarity operators that can maintain high accuracy for all data from various
imaging modalities. 2) It is not possible to estimate the upper limit these
operators can achieve and hence it is difficult to find the improvement
directions.

Image-to-image translation [16, 42, 17, 1, 21, 29, 38] based multi-modal
registration first translations multi-modal images into single-modal images
(Eq 3) using a generative adversarial network (GAN [10]), and then use Mean
Absolute Error (MAE) or Mean Squared Error (MSE) to evaluate the error at each
pixel in space (Eq 4).

$\displaystyle\min_{G}\max_{D}\mathcal{L}_{Adv}\left(G,D\right)=\mathbb{E}_{T}\left[log\left(D\left(T\right)\right)\right]+$
(3)
$\displaystyle\mathbb{E}_{M}\left[log\left(1-D\left(G\left(M\right)\right)\right)\right].$

And

$\displaystyle\hat{\theta}=\mathop{\arg\min}_{\theta}\left[\mathbb{E}_{\left(M,T\right)}\left[\|G\left(M\right),T,g_{\theta}\left(G\left(M\right),T\right)\|_{1}\right]\right].$
(4)

Image-to-image translation based registration cleverly avoids the complex
multi-modal problem and reduces the difficulty of registration to a certain
extent. However, it has obvious drawbacks. 1) The methods based on GAN require
training a generator using existing multi-modal data. The trained model will
not work if it encounters unseen data, which greatly limits its applicable
scenarios. 2) More importantly, registration is an instance problem. However,
the method based on GAN is to remap the data distribution between different
modal. As shown in Figure 1, the distribution has bias and variance. We cannot
guarantee that the translated image corresponds exactly to the instance target
image at the pixel level. For example, Figure 2 shows that there is still
residual distribution difference between the target image and translated
image. Therefore, even if they are well aligned in space, there is still a
large error in the same organ.

Figure 2: The distributions of the Moving images translated by CycleGAN still
have large residual differences from the Target images. IMSE gives smaller
error assessment values in the overlapping regions (converging to blue).

To address these challenges, we propose a novel idea based on self-
supervision, namely, Indescribable Multi-modal Spatial Evaluator, or IMSE for
short. The IMSE approach creates an accurate multi-modal spatial evaluator to
metric spatial differences between two images, and then optimizes registration
by minimizing the error predicted by the evaluator. In Figure 2, we provide a
visual demonstration of IMSE in terms of spatial location for T1-T2 and MRI-
CT, respectively. Even though distribution differences between the Moving and
Target images are still significant, IMSE is low in overlapping regions of the
same organs. The main contributions of this study can be summarized as
follows:

  * •

We introduce the concepts of relative single-modal and absolute single-modal
as an extension of the current definition of single-modality.

  * •

Based on relative single-modal and absolute single-modal, we propose a self-
supervised IMSE method to evaluate spatial differences in multi-modal image
registration. The main advantage of IMSE is that it focuses only on
differences in spatial location while ignoring differences in multi-modal
distribution caused by different image acquisition mechanisms. Our results
show that IMSE outperformed the existing metrics for registration using T1-T2
and CT-MRI datasets.

  * •

We propose a new style enhancement method named Shuffle Remap. Shuffle Remap
can help IMSE to accurate predict the difference in spatial location from an
unseen target distribution. As a enhancement method, Shuffle Remap can be
impactful in the field of domain generalization.

  * •

We develop some additional functions for IMSE. 1) As a measure, IMSE can be
integrated into both the registration based on neural network and the
traditional registration. 2) IMSE can also be used to establish a new image-
to-image translation paradigm. 3) IMSE can provide provide a convenient way to
evaluate and visualize registration results.

##  2 Related Work

Multi-modal Similarity Metric: In order to measure the spatial difference
between multi-modal images, several classical operators have been proposed.
For example, normalized cross-correlation [32, 31, 22, 15] is used to describe
the correlation between two vectors or samples of the same dimension, mutual
information [27, 23, 5, 35] is used to describe the degree of interdependence
between variables, and modal independent neighborhood descriptor [13, 8, 39,
3] is used to describe the local modal characteristics around each voxel.
These operators can also be used as loss functions in combination with neural
networks [2, 11]. The classical operators are based on mathematical knowledge
of researchers. It is difficult to apply to all modal scenes and estimate
their upper limits. As GAN [10] becomes popular in image translation task,
researchers have proposed various methods based on GAN to translate multi-
modal to single-modal images to facilitate registration. Wei et al. [37] used
CycleGAN [42] to achieve 2D MR-CT image translation, and converted the 2D
slice into 3D volumes through stacking, and finally acted on registration. Qin
Chen et al. [30] proposed an unsupervised multi-modal image-to-image synthesis
method by separating the latent shape appearance space and content information
space. Kong et al. [18] proposed a method to add correction to the image
translation to improve the quality of the translated image. These GAN based
methods remap the data distribution of different modal. There are always
biases and variances within the distribution, so registration is an instance
problem.

Domain Generalization: The goal of domain generalization (DG) is to generalize
to unseen data by training the model on source domain data [4, 6, 14, 28].
There are methods which aim to learn domain invariant representations by
minimizing domain differences between multiple source domains [9, 19, 20, 40].
In addition, several methods handle DG tasks by modifying the normalization
layer, such as instance normalization (IN) and batch normalization (BN) [7,
26, 33, 34]. In the medical field, Ziqi Zhou et al. [41] proposed the method
of Dual Normalization for segmentation. Wang et al. [36] used a domain
knowledge base to store domain specific prior knowledge, and domain attributes
to aggregate the characteristics of different domains. Shuffle Remap proposed
by us is a pure style enhancement method, which can be easily combined with
other domain generalization methods. It is well suited to situations with
large deviation scales, such as CT-MR.

##  3 Methodology

###  3.1 Motivation

In multi-modal registration, the task is greatly simplified if we can isolate
and exclude distribution differences, and focus only on spatial differences.
In fact, the GAN-based approach intends to eliminate distribution differences
by translating the source-domain image to the target-domain image. However,
although the translated image and the target image can be viewed as single-
modal relationship, the residual distribution differences may still be
significant, making MAE or MSE unsuitable for optimizing registration
performance. Further, we can define the relationship between such single-modal
images as the relative single-modal.

Figure 3: A general overview of the IMSE process. It is divided into two main
parts, the training evaluator and the training registration.

Relative single-modal: For any images $\left\\{x_{n}\right\\}_{n=1}^{n=N}$, if
all satisfy a specific and identical data distribution rule, i.e.
$\left\\{x_{n}\right\\}_{n=1}^{n=N}\sim\mathcal{D}(\mu,\,\sigma^{2})$, then
$\left\\{x_{1},...,x_{N}\right\\}$ are called relative single-modal data with
respect to each other. Relative single-modal is a abstract and extensive
concept. For example, if both CT and MR belong to the category of medical
images compared with natural images, they can be regarded as relative single-
modal images; If T1 and T2 belong to the category of MR compared with CT, they
can be regarded as relative single-modal images.

Moreover, we can define the relationship between images without residual
distribution difference as absolute single-modal.

Absolute single-modal: For any
$(x_{j},x_{k})\in\left\\{x_{n}\right\\}_{n=1}^{n=N}$, where $j\neq k$, if
$x_{j}$ and $x_{k}$ can be obtained from each other by some particular spatial
transformation $\phi$ only, i.e. $x_{j}-x_{k}\circ\phi=0$ or
$x_{k}-x_{j}\circ\phi^{-1}=0$. Then $\left\\{x_{1},...,x_{N}\right\\}$ is
called absolute single-modal data. Absolute single-modal belongs to a narrower
concept. To some extent, relative single-modal contains absolute single-modal.

The images that belong to absolute single-modal are completely consistent in
modalities. We can regard the absolute-modal difference between two images as
a spatial error. Therefore, the question becomes how to make the model measure
the spatial error from an absolute single-modal perspective.

###  3.2 IMSE

The IMSE method involves training of evaluator and registration separately. In
this section, we provide a detailed description of the method.

1.Training evaluator: In IMSE, we can completely simulate multi-modal
registration data and obtain the absolute single-modal label. We first apply
two random spatial transformations $T_{1}$ and $T_{2}$ to the original image
$x$ to obtain the transformed images $x_{1}$ and $x_{2}$, respectively. The
spatial transformation operations include overall rotation, displacement,
rescaling and random pixel-wise deformation. Since $x_{1}$ and $x_{2}$ are
from the same image $x$, they satisfy the definition of absolute single-modal
and differ only in spatial location. Once $x_{1}$ and $x_{2}$ are generated,
we subtract the two images to obtain the image of spatial position error,
which is used as the label for evaluator training. Next, we add a random noise
$\varepsilon$(see Section 3.3 for the specific noise form) to $x_{1}$ to
create distribution differences between $x_{1}$ and $x_{2}$.
$x_{1}+\varepsilon$ and $x_{2}$ are then stacked according to the number of
channels and used as inputs to the evaluator $E$. Then, the predictions of
Evaluator and previously created label are used for training (Eq. 5). It is
worth noting that the distribution of $x_{1}+\varepsilon$ is arbitrary, but
$x_{2}$ is unchanged. Therefore, the trained evaluator take the input $x_{2}$
as the reference image to predict the absolute single-modal error
($x_{2}-x_{1}$).

$\displaystyle\min_{E}\mathcal{L}_{L_{1}}\left(E\right)=\mathbb{E}_{x_{1},x_{2}}\left[\|E\left(x_{2},x_{1}+\varepsilon\right),\left(x_{2}-x_{1}\right)\|_{1}\right].$
(5)

2.Training registration: Given the moving image $x$ and the target image $y$,
deformation fields $\phi_{x2y}$ are obtained using a registration network $R$
with $x$ and $y$ as inputs. The deformation fields are used to obtain the
warped image $x\circ\phi_{x2y}$ by warping the moving image $x$. Then, the
warped image $x\circ\phi_{x2y}$ and the target images $y$ are fed into the
evaluator $E$ to get the spatial error $e$. By minimizing the error $e$, we
can optimize the parameters of the registration network $R$ (Eq. 6). We should
also use a regularization constraint based on the deformation fields (Eq. 7).
During the training of the registration network, the parameters of the
evaluator remain the same and are not updated.

$\displaystyle\min_{R}\mathcal{L}_{sim}\left(R\right)=\mathbb{E}_{x,y}\left[\|E\left(x\circ
R\left(x,y\right),y\right)\|_{1}\right].$ (6)
$\displaystyle\min_{R}\mathcal{L}_{smooth}\left(R\right)=\mathbb{E}_{x,y}\left[\|\nabla
R\left(x,y\right)\|^{2}\right].$ (7)

IMSE uses a neural network to evaluate the similarity between multi-modal
images. It is not possible to describe its computational process using
mathematical formulas like what traditional similarity operators do. This is
analogous to the discriminator in generative adversarial networks. In the
training process of the generator, it is impossible to design an analytical
operator and use it as a loss function to evaluate the authenticity of the
generated images and optimize the generator. To overcome this challenge, the
researchers use the classification loss of the discriminator to optimize the
generator indirectly. Similarly, IMSE uses the output of the evaluator to
update the registration network. Unlike GAN, IMSE has no adversarial process
and does not necessarily train both the generator and the discriminator
simultaneously as GAN does. IMSE trains the evaluator and registration network
separately. Thus, it can save more computational resources.

###  3.3 Shuffle Remap

The next question we need to consider is what factors determine the
performance limitations of IMSE. Based on the training process, IMSE belongs
to the category of self-supervision. The label used in the training is
absolutely accurate. The upper limit of performance that the evaluator can
achieve depends mainly on two factors. First, the degree of deformation added
to the image needs to sufficiently cover spatial differences presented in the
task. Second, the noise added to the image needs to have sufficient diversity
and cover the range of distribution differences.

Algorithm 1 Pseudocode of Shuffle Remap in PyTorch style.

# X: the input image and the range is [-1,1]
# r_min: Minimum number of random control points
# r_max: Maximum number of random control points

# number of randomly generated control points
$\mathrm{control\\_point{\color[rgb]{0.66,0.133,1}=}random.randint(r\\_min,r\\_max)}$
# normalize to the range of the image distribution
$\mathrm{dist{\color[rgb]{0.66,0.133,1}=}torch.rand(control\\_point){\color[rgb]{0.66,0.133,1}*}(1{\color[rgb]{0.66,0.133,1}-}({\color[rgb]{0.66,0.133,1}-}1)){\color[rgb]{0.66,0.133,1}+}({\color[rgb]{0.66,0.133,1}-}1)}$
# sort from small to large
$\mathrm{dist{\color[rgb]{0.66,0.133,1}=}torch.sort(dist)}$
# Add endpoint -1 and 1
$\mathrm{dist{\color[rgb]{0.66,0.133,1}=}torch.cat([torch.tensor([{\color[rgb]{0.66,0.133,1}-}1]),dist])}$
$\mathrm{dist{\color[rgb]{0.66,0.133,1}=}torch.cat([dist,torch.tensor([1])])}$
# shuffle the distribution and generate empty new image
$\mathrm{shuffle\\_remap{\color[rgb]{0.66,0.133,1}=}torch.randperm(control\\_point{\color[rgb]{0.66,0.133,1}+}1)}$
$\mathrm{new\\_X{\color[rgb]{0.66,0.133,1}=}torch.zeros\\_like(x)}$
for i in range(control_point +1):
$\mathrm{\qquad target\\_part{\color[rgb]{0.66,0.133,1}=}shuffle\\_remap[i]}$
$\mathrm{\qquad
min1,max1{\color[rgb]{0.66,0.133,1}=}dist[i],dist[i{\color[rgb]{0.66,0.133,1}+}1]}$
$\mathrm{\qquad
min2,max2{\color[rgb]{0.66,0.133,1}=}dist[target\\_part],dist[target\\_part{\color[rgb]{0.66,0.133,1}+}1]}$
# get the coordinates corresponding to the distribution
$\mathrm{\qquad
coord{\color[rgb]{0.66,0.133,1}=}torch.where((min1{\color[rgb]{0.66,0.133,1}<=}X){\color[rgb]{0.66,0.133,1}\&}(X{\color[rgb]{0.66,0.133,1}<}max1))}$
# Eq.(8)
$\mathrm{\qquad
new\\_X[coord]{\color[rgb]{0.66,0.133,1}=}((X[coord]{\color[rgb]{0.66,0.133,1}-}min1)/(max1{\color[rgb]{0.66,0.133,1}-}min1)){\color[rgb]{0.66,0.133,1}*}}$
$\mathrm{\qquad\qquad\qquad\qquad(max2{\color[rgb]{0.66,0.133,1}-}min2){\color[rgb]{0.66,0.133,1}+}min2}$
$\mathrm{{\color[rgb]{0,0.5,0.3215}return}\quad new\\_X}$

We propose a simple yet effective style enhancement method named Shuffle Remap
to ensure sufficient coverage of distribution differences. Specifically, we
first normalize the distribution of the original image $X$ to [-1,1], then
generate some random control points in between [-1,1] with the endpoints
$P_{0}$ and $P_{N}$ fixed at -1 and 1, respectively. These control points
randomly divide the image distribution into $N$ segments. Each segment is
assigned an index number $n$ in the order from the smallest to the largest.
After that, the segments are randomly disordered and remapped according to the
disordered order. For example, Eq. 8 demonstrates the remapping from the range
$(Pi,Pi+1)$ to the range $(Pj,Pj+1)$ for a given pixel. The complete algorithm
flow is given in Algorithm1.

$\displaystyle
x^{{}^{\prime}}=\frac{x-p_{i}}{p_{i+1}-p_{i}}*(p_{j+1}-p_{j})+p_{j}.$ (8)

Figure 4: Examples of T1 and CT after Shuffle Remap.

Figure 4 shows the effect of Shuffle Remap. We show the results for different
segments. Since the index number and location of control points are random,
the remapped images either blur the contrast (Remap.1) or enhance the contrast
(Remap.3) between the anatomical structures on the original image. The
distribution of remapped images also can be very far away (Remap.2) from the
original image distribution. In addition, with the increase of segments, the
original image can also be confused (Remap.4). Regardless of Shuffle Remap
results, the label used in evaluator training is always unique and accurate,
which has the obvious benefit of enhancing the model’s knowledge of the same
anatomical structures while reducing sensitivity to differences in image
distribution. Shuffle Remap is very different from the commonly used histogram
shift method. Histogram shift only scales the image distribution without
changing the relative relationship of the overall image distribution. Shuffle
Remap, however, is a completely random remap of the original image
distribution.

It is worth noting that we only provide a style enhancement method, and
Shuffle Remap is not irreplaceable in the IMSE architecture. Shuffle Remap is
a pure style enhancement method, which can be easily combined with other
domain generalization methods.

##  4 Experiments

In this work, we did 4 experiments to evaluate the potential of IMSE. (1) We
evaluated the performance of IMSE in multi-modal image registration and
compared it with various existing registration methods based on neural
networks. (2) We integrated IMSE into the traditional registration procedure.
(3) We investigated the feasibility of IMSE as a new image-to-image
translation paradigm. (4)We explored the accuracy of using IMSE to assess
spatial error.

###  4.1 Dataset

The first dataset is from T1-T2 modal in BraTS2019 [24], and the second
dataset is from clinical CT-MR modal. CT-MR dataset registration accuracy was
evaluated based on the parotid gland which was contoured by physicians. Table
1 provides a brief description of the datasets used in this study.

|  |  |  | 2D |  |  |  | 3D |  |
---|---|---|---|---|---|---|---|---|---|---
Source | Modality | Position | Size | Train | Test | Resize | Size | Train | Test | Resize
BraTs 2019 | T1-T2 | Brain | $240\times 240$ | 18911 | 640 | ✗ | $48\times 128\times 128$ | 187 | 20 | ✔
Clinical | CT-MR | Head neck | $192\times 192$ | 3839 | 863 | ✔ | $48\times 128\times 128$ | 80 | 18 | ✔
Table 1: A brief description of the datasets used in the study.

Moving$\rightarrow$ Target |  |  | 2D |  |  | 3D |
---|---|---|---|---|---|---|---
Methods |  Dice $\uparrow$ |  HD95 $\downarrow$ | $\|\nabla\phi\|_{2}\downarrow$ |  Dice $\uparrow$ |  HD95 $\downarrow$ | $\|\nabla\phi\|_{2}\downarrow$
T1 $\rightarrow$ T2 | Initial |  0.68 $\pm$ 0.08 |  4.17$\pm$ 1.76 | ✗ |  0.81$\pm$ 0.05 |  4.13$\pm$ 1.53 | ✗
NCC |  0.75 $\pm$ 0.05 |  2.71$\pm$ 1.33 | 0.0026 |  0.84 $\pm$ 0.03 |  3.38 $\pm$ 1.03 | 0.0036
MI |  0.82 $\pm$ 0.04 |  1.70$\pm$ 1.29 | 0.0102 |  0.86$\pm$ 0.04 |  2.79 $\pm$ 1.44 | 0.0162
MIND |  0.83 $\pm$ 0.04 |  1.66$\pm$ 1.27 | 0.0023 |  0.88$\pm$ 0.02 |  2.70 $\pm$ 1.01 | 0.0034
CycleGAN |  0.85 $\pm$ 0.03 |  1.37$\pm$ 0.95 | 0.0061 |  0.88$\pm$ 0.02 |  2.94 $\pm$ 0.93 | 0.010
RegGAN |  0.86 $\pm$ 0.03 |  1.25 $\pm$ 0.90 | 0.0091 |  0.89$\pm$ 0.01 |  2.85 $\pm$ 0.83 | 0.0090
IMSE(BC) |  0.83 $\pm$ 0.04 |  1.72 $\pm$ 1.30 | 0.0125 |  0.84$\pm$ 0.04 |  3.21 $\pm$ 1.33 | 0.0105
IMSE(SR) | 0.89 $\pm$ 0.02 | 1.06$\pm$ 0.87 | 0.0023 | 0.91$\pm$ 0.01 | 2.36 $\pm$ 0.77 | 0.0032
T2 $\rightarrow$ T1 | NCC |  0.74 $\pm$ 0.04 |  2.76$\pm$ 1.35 | 0.0026 |  0.84$\pm$ 0.03 |  3.07 $\pm$ 1.04 | 0.0038
MI |  0.79 $\pm$ 0.05 |  1.58$\pm$ 1.31 | 0.0103 |  0.88$\pm$ 0.04 |  2.33 $\pm$ 1.41 | 0.0166
MIND |  0.81 $\pm$ 0.03 |  2.15$\pm$ 1.28 | 0.0023 |  0.88$\pm$ 0.02 |  2.42 $\pm$ 1.05 | 0.0035
CycleGAN |  0.86 $\pm$ 0.04 |  1.19$\pm$ 0.94 | 0.0056 |  0.88$\pm$ 0.02 |  2.95 $\pm$ 0.95 | 0.009
RegGAN |  0.86 $\pm$ 0.03 |  1.20$\pm$ 0.90 | 0.0071 |  0.89$\pm$ 0.01 |  2.71 $\pm$ 0.80 | 0.0085
IMSE(BC) |  0.80 $\pm$ 0.04 |  1.87 $\pm$ 1.34 | 0.0122 |  0.85$\pm$ 0.03 |  3.01 $\pm$ 1.26 | 0.0097
IMSE(SR) | 0.89 $\pm$ 0.02 | 0.85$\pm$ 0.86 | 0.0022 | 0.91$\pm$ 0.01 | 2.21 $\pm$ 0.75 | 0.0025

Moving$\rightarrow$ Target |  |  | 2D |  |  | 3D |
---|---|---|---|---|---|---|---
Methods |  Dice $\uparrow$ |  HD95 $\downarrow$ | $\|\nabla\phi\|_{2}\downarrow$ |  Dice $\uparrow$ |  HD95 $\downarrow$ | $\|\nabla\phi\|_{2}\downarrow$
CT $\rightarrow$ MR | Initial |  0.40 $\pm$ 0.07 |  10.05$\pm$ 1.93 | ✗ |  0.48$\pm$ 0.05 |  5.64$\pm$ 1.41 | ✗
NCC |  0.49 $\pm$ 0.04 |  9.05$\pm$ 1.75 | 0.0074 |  0.54 $\pm$ 0.03 |  5.11 $\pm$ 1.18 | 0.017
MI |  0.50 $\pm$ 0.05 |  8.90$\pm$ 1.77 | 0.0075 |  0.55$\pm$ 0.03 |  5.14 $\pm$ 1.16 | 0.019
MIND |  0.50 $\pm$ 0.04 |  8.55$\pm$ 1.79 | 0.0019 |  0.54$\pm$ 0.02 |  5.10 $\pm$ 1.07 | 0.008
CycleGAN |  0.56 $\pm$ 0.04 |  8.01$\pm$ 1.69 | 0.0022 |  0.58$\pm$ 0.02 |  4.62 $\pm$ 0.95 | 0.010
RegGAN |  0.57 $\pm$ 0.03 |  7.83 $\pm$ 1.63 | 0.0020 |  0.60$\pm$ 0.01 |  4.45 $\pm$ 0.90 | 0.009
IMSE(BC) |  0.50 $\pm$ 0.04 |  8.63 $\pm$ 1.69 | 0.0072 |  0.55$\pm$ 0.03 |  5.08 $\pm$ 1.06 | 0.019
IMSE(SR) | 0.61 $\pm$ 0.02 | 6.92$\pm$ 1.51 | 0.0017 | 0.62$\pm$ 0.01 | 4.23 $\pm$ 0.82 | 0.007
MR $\rightarrow$ CT | NCC |  0.49 $\pm$ 0.05 |  9.04$\pm$ 1.75 | 0.0070 |  0.57$\pm$ 0.03 |  5.01 $\pm$ 1.09 | 0.013
MI |  0.50 $\pm$ 0.04 |  9.18$\pm$ 1.75 | 0.0076 |  0.58$\pm$ 0.02 |  5.13 $\pm$ 1.15 | 0.019
MIND |  0.51 $\pm$ 0.04 |  8.86$\pm$ 1.78 | 0.0019 |  0.56$\pm$ 0.03 |  5.05 $\pm$ 1.19 | 0.007
CycleGAN |  0.58 $\pm$ 0.04 |  7.56$\pm$ 1.66 | 0.0018 |  0.58$\pm$ 0.01 |  4.81 $\pm$ 0.89 | 0.011
RegGAN |  0.59 $\pm$ 0.02 |  7.30$\pm$1.61 | 0.0021 |  0.61$\pm$ 0.01 |  4.62 $\pm$ 0.89 | 0.009
IMSE(BC) |  0.50 $\pm$ 0.04 |  8.85 $\pm$ 1.71 | 0.0069 |  0.56$\pm$ 0.04 |  4.97 $\pm$ 1.16 | 0.016
IMSE(SR) | 0.60 $\pm$ 0.01 | 7.26$\pm$ 1.58 | 0.0020 | 0.62$\pm$ 0.01 | 4.45 $\pm$ 0.86 | 0.008

Table 2: Registration results of various methods based on the T1-T2 and CT-MR
dataset. Initial indicates the results before registration. The source data
used to train the IMSE were T1 and CT.

###  4.2 IMSE for Registration Based on Neural Network

In this subsection, we compared various registration methods based on neural
networks. Baseline uses traditional similarity operators as loss functions to
update the registration network, including NCC [22], MI [27], and MIND [13].
There are also GAN based methods which do translation first and then use MAE
as the loss function of the registered network, including CycleGAN [42],
RegGAN [17]. We also compared histogram shift using the Bézier curve [25]
(IMSE (BC)) and Shuffle Remap (IMSE (SR)) for style enhancement. The random
range of N in Shuffle Remap is [2, 50].

Figure 5: Registration results for various registration methods. Four rows
correspond to T1$\rightarrow$T2, T2$\rightarrow$T1, CT$\rightarrow$MR and
MR$\rightarrow$CT registrations, respectively. Orange contours are based on T1
and CT images. Red contours are based on T2 and MR images. The first column
(Moving) shows the contour difference without registration.

For a fair comparison, all methods used a unified registration network
model–VoxelMorph [2]. Please note that in the two datasets, the source data
used to train the estimator were T1 and CT, respectively. The network
structure adopted by the evaluator is ResNet [12], which is consistent with
the generator used in CycleGAN and RegGAN. We added random affine and non-
affine transformations to the moving and target images during training and
testing, including angular rotations of [-3,3], displacement of [-8%,+ 8%],
scaling of [8%,+8%]. The non-affine transformation was generated by spatially
transforming the moving and target images using elastic transformations
followed by Gaussian smoothing. The degree of deformation was 80 and the
radius of Gaussian smoothing was 12.

Registration results of various methods based on T1-T2 and MR-CT datasets were
summarized in Table 2. We performed both forward and reverse registration for
each dataset. Registration performance was measured using Dice, Hausdorff
distances, and the smoothness which was defined by the average gradient of the
deformation field (in the case of 2D, we only counted the slices that
contained contours). For T1-T2 and MR-CT datasets, IMSE achieved the best
results based on all metrics, both in 2D and 3D. Because the registration
network used by all methods was the same, the influence of the model was
excluded. IMSE had both high registration accuracy and a smoother deformation
field, which in general is difficult to achieve simultaneously. It was likely
due to the accurate estimate of spatial errors with the adoption of the
estimator, which allowed the registration model to achieve a better alignment
at a small deformation cost. In addition, our results show that when the
Bézier curve was used for data enhancement, it has no performance advantage
compared with other methods. But the combination of IMSE and Shuffle Remap
provided the best performance among all registration methods. We want to
further explore how different segments N of shuffle remap affect the
registration results. As shown in Table 3, we can first see that with the
increase of N, the performance will improve. In addition, for T1-T2, there is
no significant difference between N of 30 and 50. For MR-CT, a larger N
obviously brings better scores since MR-CT shows larger distribution
differences than T1-T2, which requires more complex style enhancement.

T2 $\rightarrow$ T1 | N=2 | N=[2,10] | N=[2,30] | N=[2,50]
---|---|---|---|---
Dice | 0.85 | 0.88 | 0.89 | 0.89
HD95 | 1.21 | 0.93 | 0.88 | 0.85
$\|\nabla\phi\|_{2}$ | 0.0028 | 0.0024 | 0.0018 | 0.0022

MR $\rightarrow$ CT | N=2 | N=[2,10] | N=[2,30] | N=[2,50]
---|---|---|---|---
Dice | 0.51 | 0.55 | 0.57 | 0.60
HD95 | 8.83 | 8.21 | 7.49 | 7.26
$\|\nabla\phi\|_{2}$ | 0.020 | 0.0029 | 0.0025 | 0.0020

Table 3: In 2D case, shuffle remap adopts different parameter N

###  4.3 IMSE for Traditional Registration

Compared to the existing deep learning registration methods which directly
provide deformation fields, IMSE essentially evaluates the similarity between
multi-modal images through neural networks. Then, the neural network can
accurately achieve backward propagation. Therefore, IMSE can be readily
integrated into the traditional registration process, such as replacing the
$L_{sim}$ function in Eq 1 with a trained evaluator. By first initializing a
deformation field of 0 and then optimizing it through similarity loss
(Similarity operator or IMSE) and regularization loss(Eq 7). We set the
deformation field size to [64,128,128], the learning rate to 1, the number of
iterations to 200, and the optimizer to $adam$. In this section, we not only
evaluate the traditional multi-modal registration, but also compare IMSE with
single-modal registration using MAE as an optimization measure. Single-modal
registration is evaluated using T1$\rightarrow$T1 and CT$\rightarrow$CT data.
Since the single-modal images were from different patients with spatial
differences significantly larger than those from the same patient, we screened
5 patients whose spatial location differences were relatively small to mimic
the scenario of registering images from the same patient. The results are
shown in Table 4. As the metric for registration optimization, IMSE still
achieved the best performance in multi-modal conditions. We focus on the
results of single-modal registration. IMSE performed much better than MAE in
all aspects. This is because even in single-modal datasets, there are still
residual distribution differences. T1-T1 or CT-CT should be categorized as
relatively single-modal data, especially when images from different patients
are registered to each other. MAE cannot ignore residual distribution
differences whereas IMSE can.

Moving$\rightarrow$Target | Methods | Dice $\uparrow$ | HD95 $\downarrow$ | $\|\nabla\phi\|_{2}\downarrow$
---|---|---|---|---
T2 $\rightarrow$ T1 | NCC | 0.84 $\pm$ 0.03 | 3.40$\pm$ 1.14 | 0.016
MI | 0.85 $\pm$ 0.03 | 3.26$\pm$ 1.22 | 0.06
MIND | 0.89 $\pm$ 0.02 | 3.01$\pm$ 1.16 | 0.002
IMSE | 0.91$\pm$ 0.01 | 2.62$\pm$ 0.81 | 0.002
MR $\rightarrow$ CT | NCC | 0.54 $\pm$ 0.03 | 5.18$\pm$ 1.20 | 0.011
MI | 0.55 $\pm$ 0.02 | 5.29$\pm$ 1.18 | 0.05
MIND | 0.55 $\pm$ 0.02 | 4.93$\pm$ 1.24 | 0.008
IMSE | 0.61$\pm$ 0.01 | 4.37$\pm$ 0.81 | 0.007
T1 $\rightarrow$ T1 | MAE | 0.64 $\pm$ 0.03 | 8.38$\pm$ 1.43 | 0.018
IMSE | 0.67 $\pm$ 0.02 | 7.4$\pm$ 1.22 | 0.005
CT $\rightarrow$ CT | MAE | 0.56 $\pm$ 0.02 | 4.92$\pm$ 1.07 | 0.012
IMSE | 0.59 $\pm$ 0.02 | 4.80$\pm$0.87 | 0.005
Table 4: Comparison of registration results for traditional algorithms.

###  4.4 IMSE for Image-to-image Translation

To explain how IMSE may enable a new paradigm of image-to-image translation,
we use multi-modal images $x$ and $y$ as an example where $x$ is the reference
image and $y$ is the image awaiting to be translated. Input $x$ and $y$ into
IMSE, and IMSE will use x as the reference image to predict the absolute
single-modal error:$E(x,y)\approx x-x^{{}^{\prime}}$. Where, the distribution
of $x$ and $x^{{}^{\prime}}$ is consistent. By subtracting $E(x,y)$ from the
reference image $x$, we can get the translated image $x^{{}^{\prime}}$ , i.e.,
$x^{{}^{\prime}}\approx x-E(x,y)$. In the new paradigm, image-to-image
translation is achieved by reverse inference based on the prediction of the
evaluator. In Figure 6, we show a few examples of using IMSE to perform image-
to-image translation. $x^{{}^{\prime}}$ is the modal translated image of $y$
as it has the spatial characteristics of $y$ but the modal characteristics of
$x$. For example, the dental artifact in the reference CT image in Figure 6
remains in the translated image. This also verifies that IMSE is an instance
mapping method based on absolute single-modal.

Figure 6:  IMSE is used for image to image translation. Where, $y$ is the source image, $x$ is different reference images, $E(x,y)$ is the prediction result of IMSE and $x^{{}^{\prime}}=x-E(x,y)$. Source$\rightarrow$Target | Methods | NMAE $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---|---|---
T2 $\rightarrow$ T1 | CycleGAN | 0.088 | 24.1 | 0.89
RegGAN | 0.071 | 25.5 | 0.90
IMSE | 0.029 | 91.6 | 0.96
MR $\rightarrow$ CT | CycleGAN | 0.049 | 22.9 | 0.88
RegGAN | 0.041 | 24.1 | 0.89
IMSE | 0.022 | 41.3 | 0.93
Table 5: The results of image-to-image translation.

Table 5 shows the comparison results between IMSE and baseline methods.
Spatial transformations were added to $x$ and $y$ in IMSE to prevent alignment
between input images. Despite this, IMSE still produces superior results
compared to other methods due to the use of an additional target reference
image. Thus, the comparison between IMSE and GAN-based methods seems unjust.
We aim to investigate the contrasting use scenarios of these two methods,
further.

IMSE based image-to-image translation differs from GAN based translation in
two aspects. 1) The result of image-to-image translation based on GAN is
unique whereas IMSE based depends on the characteristics of the reference
image. 2) GAN based image-to-image translation requires two modal data whereas
IMSE based translation only uses one modal data for training. IMSE is very
promising for medical image-to-image translation. It requires less data for
training and therefore can reduce expensive data costs. It is better suited to
the complexity and diversity of medical scenes. Also, medical image
translation requires high accuracy. It is always beneficial to ensure that the
characteristics of the translated image come from the desired reference image.

###  4.5  IMSE for Spatial Error

Without labels, accurate evaluation of registration results is always
challenging. IMSE has great potential in offering an objective metric to
accurately evaluate registration performance. Since the output of IMSE has the
same size as the input image, it can provide pixel-wise registration error
estimation. In Figure 7, we demonstrate spatial errors estimated by the
CycleGAN, RegGAN and IMSE methods in both misaligned and aligned cases.
CycleGAN and RegGAN translated the moving image first and then calculate MAE
between the translated image and the target image.

The estimated spatial errors in the CycleGAN and RegGAN methods could not
exclude the effect of residual distribution between the translated and target
images. The errors remained significant even if images were well aligned. As a
comparison, IMSE reported large spatial errors in misaligned regions but small
spatial errors in aligned regions.

Figure 7:  Demonstration of errors estimated by the CycleGAN, RegGAN and IMSE
methods in both misaligned and aligned cases.

Figure 8:  Correlation of estimated spatial errors with Dice for the IMSE,
CycleGAN, RegGAN methods.

To quantitatively demonstrate the potential of IMSE for registration
performance evaluation, we explored the correlation between IMSE output and
Dice. Based on the joint region of Moving and Target masks, we calculated the
mean value of IMSE outputs. Then we subtracted the mean absolute value from 1
and performed a normalization to get the value which indicated registration
performance for IMSE. Similarly, we calculated values indicating registration
performance for CycleGAN and RegGAN. The tests were performed in 3D due to
limited test data. To increase the number of test data, we simulated more test
data through many random transformations of the image. Figure 8 clearly
demonstrates the positive correlation between IMSE and Dice, which confirms
the potential of using IMSE to accurately evaluate spatial errors. As a
comparison, CycleGAN and RegGAN do not have obvious correlation with Dice,
especially on the T1-T2 dataset.

##  5 Discussion

In this study, we propose a new approach IMSE for multi-modal image
registration. IMSE is simple yet powerful. It can be used as a training
registration model. As a metric, IMSE can be used to evaluate the registration
results or be combined with traditional registration process. It can also be
used to perform image-to-image translation. IMSE uses neural networks instead
of similarity operators as loss functions to achieve better results. It also
stimulates thoughts of using neural network as indirect constraints to solve
challenging problems, instead of committing significant time and efforts
searching for operators or loss functions for better performance. Shuffle
Remap is an essential component of IMSE. It greatly reduces the amount of data
required for model training. All trained models in the current study used only
one modal of data and the evaluator has the capability to evaluate and
translation unseen data as well. Even though our study focused on medical
images, the principle of IMSE should apply to natural images as well. In the
future, we will continue investigating IMSE from three aspects. 1) We have
demonstrated the correlation between IMSE and Dice. But the correlation is not
as strong as we prefer. We will explore various ways of calculating spatial
errors to see if it is possible to improve correlation. 2) The evaluator is
based on neural networks. If it is integrated into the traditional algorithm,
backward propagation requires more time. We will explore whether the evaluator
can achieve similar results using a simpler architecture. 3) We will explore
other variants of Shuffle Remap and prove that Shuffle Remap as a method of
style enhancement can be more impactful in domain generalization.

## References

  * [1] Moab Arar, Yiftach Ginger, Dov Danon, Amit H Bermano, and Daniel Cohen-Or.  Unsupervised multi-modal image registration via geometry preserving image-to-image translation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13410–13419, 2020.
  * [2] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca.  Voxelmorph: a learning framework for deformable medical image registration.  IEEE transactions on medical imaging, 38(8):1788–1800, 2019.
  * [3] Thuvanan Borvornvitchotikarn and Werasak Kurutach.  mirid: Multi-modal image registration using modality-independent and rotation-invariant descriptor.  Symmetry, 12(12):2078, 2020.
  * [4] Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman.  Learning to balance specificity and invariance for in and out of domain generalization.  In European Conference on Computer Vision, pages 301–318. Springer, 2020.
  * [5] Bob D de Vos, Bas HM van der Velden, Jörg Sander, Kenneth GA Gilhuijs, Marius Staring, and Ivana Išgum.  Mutual information for unsupervised deep learning image registration.  In Medical Imaging 2020: Image Processing, volume 11313, pages 155–161. SPIE, 2020.
  * [6] Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao.  Learning to learn with variational information bottleneck for domain generalization.  In European Conference on Computer Vision, pages 200–216. Springer, 2020.
  * [7] Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou.  Adversarially adaptive normalization for single domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8208–8217, 2021.
  * [8] Yabo Fu, Yang Lei, Tonghe Wang, Jun Zhou, Walter J Curran, Pretesh Patel, Tian Liu, and Xiaofeng Yang.  Deformable mri-ct liver image registration using convolutional neural network with modality independent neighborhood descriptors.  In Medical Imaging 2021: Computer-Aided Diagnosis, volume 11597, pages 102–107. SPIE, 2021.
  * [9] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi.  Domain generalization for object recognition with multi-task autoencoders.  In Proceedings of the IEEE international conference on computer vision, pages 2551–2559, 2015.
  * [10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [11] Courtney K Guo.  Multi-modal image registration with unsupervised deep learning.  PhD thesis, Massachusetts Institute of Technology, 2019.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] Mattias P Heinrich, Mark Jenkinson, Manav Bhushan, Tahreema Matin, Fergus V Gleeson, Michael Brady, and Julia A Schnabel.  Mind: Modality independent neighbourhood descriptor for multi-modal deformable registration.  Medical image analysis, 16(7):1423–1435, 2012.
  * [14] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry.  Augment your batch: Improving generalization through instance repetition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8129–8138, 2020.
  * [15] Chongfei Huang, Chenhui Qiu, Zhiyi Peng, Jing Yuan, and Dexing Kong.  Iterative reweighted local cross correlation method for nonlinear registration of multiphase liver ct images.  In 2021 IEEE International Conference on Image Processing (ICIP), pages 136–140. IEEE, 2021.
  * [16] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [17] Lingke Kong, Chenyu Lian, Detian Huang, Yanle Hu, Qichao Zhou, et al.  Breaking the dilemma of medical image-to-image translation.  Advances in Neural Information Processing Systems, 34:1964–1978, 2021.
  * [18] Lingke Kong, Chenyu Lian, Detian Huang, Yanle Hu, Qichao Zhou, et al.  Breaking the dilemma of medical image-to-image translation.  Advances in Neural Information Processing Systems, 34:1964–1978, 2021.
  * [19] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot.  Domain generalization with adversarial feature learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400–5409, 2018.
  * [20] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao.  Deep domain generalization via conditional invariant adversarial networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 624–639, 2018.
  * [21] Chenyu Lian, Xiaomeng Li, Lingke Kong, Jiacheng Wang, Wei Zhang, Xiaoyang Huang, and Liansheng Wang.  Cocyclereg: Collaborative cycle-consistency method for multi-modal medical image registration.  Neurocomputing, 2022.
  * [22] Shan Liu, Bo Yang, Yang Wang, Jiawei Tian, Lirong Yin, and Wenfeng Zheng.  2d/3d multimode medical image registration based on normalized cross-correlation.  Applied Sciences, 12(6):2828, 2022.
  * [23] Frederik Maes, Andre Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens.  Multimodality image registration by maximization of mutual information.  IEEE transactions on Medical Imaging, 16(2):187–198, 1997.
  * [24] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
  * [25] Michael E Mortenson.  Mathematics for computer graphics applications.  Industrial Press Inc., 1999.
  * [26] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang.  Two at once: Enhancing learning and generalization capacities via ibn-net.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 464–479, 2018.
  * [27] Josien PW Pluim, JB Antoine Maintz, and Max A Viergever.  Mutual-information-based registration of medical images: a survey.  IEEE transactions on medical imaging, 22(8):986–1004, 2003.
  * [28] Fengchun Qiao, Long Zhao, and Xi Peng.  Learning to learn single domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12556–12565, 2020.
  * [29] Chen Qin, Bibo Shi, Rui Liao, Tommaso Mansi, Daniel Rueckert, and Ali Kamen.  Unsupervised deformable registration for multi-modal images via disentangled representations.  In International Conference on Information Processing in Medical Imaging, pages 249–261. Springer, 2019.
  * [30] Chen Qin, Bibo Shi, Rui Liao, Tommaso Mansi, Daniel Rueckert, and Ali Kamen.  Unsupervised deformable registration for multi-modal images via disentangled representations.  In International Conference on Information Processing in Medical Imaging, pages 249–261. Springer, 2019.
  * [31] Y Raghavender Rao, Nikhil Prathapani, and E Nagabhooshanam.  Application of normalized cross correlation to image registration.  International Journal of Research in Engineering and Technology, 3(5):12–16, 2014.
  * [32] Jignesh N Sarvaiya, Suprava Patnaik, and Salman Bombaywala.  Image registration by template matching using normalized cross-correlation.  In 2009 international conference on advances in computing, control, and telecommunication technologies, pages 819–822. IEEE, 2009.
  * [33] Mattia Segu, Alessio Tonioni, and Federico Tombari.  Batch normalization embeddings for deep domain generalization.  Pattern Recognition, page 109115, 2022.
  * [34] Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han.  Learning to optimize domain specific normalization for domain generalization.  In European Conference on Computer Vision, pages 68–83. Springer, 2020.
  * [35] Gerard Snaauw, Michele Sasdelli, Gabriel Maicas, Stephan Lau, Johan Verjans, Mark Jenkinson, and Gustavo Carneiro.  Mutual information neural estimation for unsupervised multi-modal registration of brain images.  arXiv preprint arXiv:2201.10305, 2022.
  * [36] Shujun Wang, Lequan Yu, Kang Li, Xin Yang, Chi-Wing Fu, and Pheng-Ann Heng.  Dofe: Domain-oriented feature embedding for generalizable fundus image segmentation on unseen datasets.  IEEE Transactions on Medical Imaging, 39(12):4237–4248, 2020.
  * [37] Dongming Wei, Sahar Ahmad, Jiayu Huo, Wen Peng, Yunhao Ge, Zhong Xue, Pew-Thian Yap, Wentao Li, Dinggang Shen, and Qian Wang.  Synthesis and inpainting-based mr-ct registration for image-guided thermal ablation of liver tumors.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 512–520. Springer, 2019.
  * [38] Zhe Xu, Jie Luo, Jiangpeng Yan, Ritvik Pulya, Xiu Li, William Wells, and Jayender Jagadeesan.  Adversarial uni-and multi-modal stream networks for multimodal image registration.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 222–232. Springer, 2020.
  * [39] Feng Yang, Mingyue Ding, and Xuming Zhang.  Non-rigid multi-modal 3d medical image registration based on foveated modality independent neighborhood descriptor.  Sensors, 19(21):4675, 2019.
  * [40] Sergey Zakharov, Wadim Kehl, and Slobodan Ilic.  Deceptionnet: Network-driven domain randomization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 532–541, 2019.
  * [41] Ziqi Zhou, Lei Qi, Xin Yang, Dong Ni, and Yinghuan Shi.  Generalizable cross-modality medical image segmentation via style augmentation and dual normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20856–20865, 2022.
  * [42] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.