# A Maturity Model for Operations in Neuroscience Research

Erik C. Johnson1,a, Thinh T. Nguyen2, Benjamin K. Dichter3, Frank Zappulla4,
Montgomery Kosma2,
Kabilar Gunalan2,5, Yaroslav O. Halchenko6, Shay Q. Neufeld7, Michael
Schirner8-12, Petra Ritter8-12,
Maryann E. Martone13, Brock Wester1, Franco Pestilli14, Dimitri Yatsenko2,b
1 Research and Exploratory Development Department,
Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA
2 DataJoint, Houston, TX, USA
3 CatalystNeuro, Benicia, CA, USA
4 Digital R&D Creation Center, Pfizer Inc., USA
5 McGovern Institute for Brain Research, Massachusetts Institute of
Technology, Cambridge, MA, USA
6 Center for Open Neuroscience, Department of Psychological and Brain
Sciences,
Dartmouth College, New Hampshire, USA
7 Inscopix, a Bruker company, Mountain View, CA, USA
8 Berlin Institute of Health (BIH) at Charité – Universitätsmedizin Berlin,
Berlin, Germany
9 Department of Neurology with Experimental Neurology, Charité,
Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin
and Humboldt Universität zu Berlin, Berlin, Germany
10 Bernstein Focus State Dependencies of Learning and
Bernstein Center for Computational Neuroscience, Berlin, Germany
11 Einstein Center for Neuroscience Berlin, Berlin, Germany
12 Einstein Center Digital Future, Berlin, Germany
13 Department of Neurosciences, University of California San Diego, La Jolla,
CA, USA
14 Department of Psychology, University of Texas Austin, Austin, TX, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Scientists are adopting new approaches to scale up their activities and goals.
Progress in neurotechnologies, artificial intelligence, automation, and tools
for collaboration promises new bursts of discoveries. However, compared to
other disciplines and the industry, neuroscience laboratories have been slow
to adopt key technologies to support collaboration, reproducibility, and
automation. Drawing on progress in other fields, we define a roadmap for
implementing automated research workflows for diverse research teams. We
propose establishing a five-level capability maturity model for operations in
neuroscience research. Achieving higher levels of operational maturity
requires new technology-enabled methodologies, which we describe as “SciOps”.
The maturity model provides guidelines for evaluating and upgrading operations
in multidisciplinary neuroscience teams.

_Keywords_ DevOps $\cdot$ DataOps $\cdot$ MLOps $\cdot$ SciOps $\cdot$
Capability Maturity Model $\cdot$ neuroscience $\cdot$ operations research
$\cdot$ open science $\cdot$ closed-loop experiments $\cdot$ digital twin
$\cdot$ FAIR $\cdot$ reproducible research $\cdot$ automated workflows $\cdot$
artificial intelligence

##  1 Scaling operations in neuroscience research

Neuroscience, driven by technological advancement, stands on the cusp of
groundbreaking discoveries. Current techniques enable the capture and
modulation of intact neural circuit activity during complex behaviors,
aligning these recordings with detailed molecular and anatomical maps.
Advanced statistical tools unlock computational patterns in this vast data.
Neuroscientists are increasingly coming together in large teams to develop and
implement these new techniques at scales beyond the individual laboratory.
However, larger projects often undergo chaotic phases as diverse teams strive
to integrate disparate technical and organizational methods. The central
question looms: How can advancements in neurotechnologies be translated into
substantial scientific breakthroughs?

To keep pace with neurotechnologies, research teams require a revolution in
their scientific operations. As the complexity of coordinating data and
computation grows, it is crucial to boost collaboration, streamline processes,
reduce errors, promote data integration and prioritize reproducibility. A new
catalyst for discovery is the integration of artificial intelligence into
research activities—a prospect that demands a formalized approach to
scientific workflows1, 2. Through enhanced operations, teams can integrate and
navigate vast data, blending human ingenuity with emerging artificial
intelligence. As the challenges increase, a clear roadmap becomes vital,
steering research teams from their current practices towards greater
capacities for ambitious projects.

##  2 Capability Maturity Model for Scientific Operations

To address the question of research teams’ readiness to tackle complex
challenges with robust methods, we introduce the Capability Maturity Model for
Scientific Operations (Fig. 1). Capability Maturity Models (CMMs) have a well-
established history in engineering and related fields, providing structured
frameworks for assessing and enhancing processes within organizations3. These
frameworks are commonly employed by corporations and governments to evaluate
contractors and strategically plan for the future. Enhancing operational
maturity equips teams to embark on ambitious initiatives aimed at solving
complex problems.

Our inspiration for this model derives from the CMMI (Capability Maturity
Model Integration) family of models, initially developed by the Software
Engineering Institute at Carnegie Mellon University for software engineering
and subsequently adapted to various industries4. Drawing on our extensive
experience in coordinating large-scale research collaborations in
neuroscience, we have tailored key concepts from CMMI to the unique challenges
and opportunities presented in current neuroscience projects. While our
primary focus remains on experimental neuroscience, the core principles we
define here can apply broadly in other disciplines.

Our model places research teams into five maturity levels based on their
approach to planning and executing essential activities. This categorization
includes multiple criteria: team structure, formal processes, training,
software development, data management, and computational infrastructure and
procedures. The model can act as a step-by-step guide, assisting teams in
identifying essential steps to expand their capabilities.

Figure 1: The Capability Maturity Model for Scientific Operations in
neuroscience research (Neuro SciOps CMM v1.0). We define SciOps as a set
methodologies for implementing advanced capabilities.

Today, most neuroscience teams hover between Levels 1 and 2, though some
specialization occurs among subfields. For instance, scientific operations in
human neuroimaging tend to be more mature than in experimental
neurophysiology5. Funding policies and publisher mandates for open data and
reproducible results are nudging teams towards Level 3, which emphasizes
community standards. By Level 4, teams prioritize research automation,
scalable computing, and efficient team workflows. This approach is most
commonly achieved in larger, centralized institutions, but can be adapted to
smaller teams through better tools and platforms. However, the peak of Level
5, promising a significant leap in discoveries by closing the discovery loop
with the help of artificial intelligence, remains a goal yet to be achieved by
any team. We delve into each level in the sections below.

###  2.1 Level 1: Initial

At the outset of scientific endeavors, such as establishing a new laboratory,
embarking on a Ph.D. project, or initiating a novel experiment, teams
typically find themselves at Level 1 of the maturity model. This stage is
characterized by a high degree of flexibility and customization in
experimental and analytical methods. Data volumes at Level 1 are typically
limited, not requiring high throughput. Each project adopts tailored
approaches, with custom software and manual data management on dedicated lab
infrastructure. Standardized methods are largely absent at this stage.

Experimentation without rigid methods, often described as “tinkering,” has
always been foundational to research6. Remarkable achievements, including
those recognized with Nobel Prizes, have commonly emerged from Level 1
practices. Nevertheless, in the context of today’s technology-driven and data-
intensive projects, there is a growing need for more organized, open, and
reproducible methods. Level 1 practices make it difficult for individual labs
to adapt their processes for larger collaborations. Operational maturity
should align with the complexity and scope of the research being conducted,
much like how biosafety levels are established to match research demands.

While maintaining space for intuition-driven and flexible activities is
crucial at all maturity levels, successful methods and practices can be
standardized and scaled up as research demands increase. Moreover, a team’s
maturity level is not necessarily a reflection of its size or its funding
level. The ultimate aim of this model is to provide research teams with the
necessary tools and guidance to elevate their operational maturity to meet the
evolving demands of their research.

###  2.2 Level 2: Managed

To expand their scope and tackle larger, more complex projects, research teams
must progress to Level 2, where the emphasis shifts to developing lab-wide
standards. These standards enhance consistency and predictability in internal
operations, thereby facilitating more effective teamwork and leading to more
credible findings.

Key characteristics of Level 2 teams include:

  1. 1.

Repeatable Processes: Laboratories establish uniform methods applicable across
various projects. This standardization extends to data management with shared
storage, standardized formats, and structured naming conventions, ensuring
data integrity. Software practices also evolve, incorporating version control,
documentation, testing, and code review processes 7, 8, 5. A notable practice
at this level is the development and maintenance of a stable data pipeline,
optimized for efficient, long-term use.

  2. 2.

Role Specialization: Level 2 teams clearly define team roles and
responsibilities. This specialization allows for an efficient distribution of
tasks and maximizes the use of individual expertise. A common practice is
rotating trainees through different roles to provide a comprehensive
understanding of lab operations.

  3. 3.

Quality Control: Level 2 introduces rigorous procedures for continuously
monitoring and validating the accuracy and reliability of experimental
results. These include instrument calibration, software testing, and signal
quality assessment. Quality control criteria are established and periodically
updated to ensure the highest standards.

  4. 4.

Training Programs: Level 2 teams provide a structured onboarding process for
new members as well as ongoing training initiatives. These programs aim to
keep the team proficient in lab operations and abreast of the latest
developments. Some labs extend their training efforts beyond their teams,
hosting seminars and workshops for external groups to disseminate their
standardized practices.

The progression to Level 2 marks a significant step towards operational
maturity, characterized by a systematic approach to research, a focus on
quality and reliability, and an emphasis on continuous learning and
improvement. However, uniting multiple Level 2 labs within a consortium for
collaborative projects poses significant challenges owing to mismatching
processes employed in each lab.

###  2.3 Level 3: Defined

Level 3 marks a stage where research teams embrace practices conducive to
robust collaborations spanning various laboratories and disciplines. Level 3
labs excel in joining forces within multilab consortia, streamlining
interaction, and harmonizing data processes. Key features of this level
include the adoption of common data standards and interoperable computational
frameworks, essential for efficient use and sharing of data and software among
teams.

Key characteristics of Level 3 teams are:

  1. 1.

Open-Source Ecosystems: Level 3 teams are deeply engaged with resilient open-
source software. These ecosystems are not just about providing tools that are
“free to use” but also involve responsive community governance setting
standards for quality, reliability, and reproducibility. They form the
foundation for state-of-the-art projects and offer community support and
educational resources. Level 3 teams align their work with these community-
driven open-source endeavors, promoting consistency, integration, reliability,
reproducibility, and sustainability.

These teams adopt disciplined practices for software management, including
principled code management, peer review, validation, and testing. This
approach enables the creation of evolving data pipelines and computational
workflows with minimal downtime, ensuring continuous research progression.

  2. 2.

FAIR Data: Level 3 teams develop, adopt, and promote harmonized initiatives
for data standards, fostering interoperability of tools and processes across
research groups. This includes adherence to the FAIR principles (Findable,
Accessible, Interoperable, and Reusable) for scientific data9. Such standards
facilitate the exchange and reuse of complex data, supported by robust systems
including data sharing platforms that enable reproducibility and re-analysis.
Examples in neuroscience include data standards around the BIDS10 and NWB
projects11.

Data exchange and reuse requires infrastructure. Robust data sharing platforms
and collaborative research environments are set up to facilitate joint
research endeavors. Neuroscience data archives such as DANDI12, BrainLife13,
BossDB14, and OpenNeuro15 not only store the data but also facilitate
reproducibility and new analysis. Distributed data management systems such as
DataLad, which relies on git-annex, facilitate not only versioning of data but
also unified data access and exchange across multiple work sites and
archives16, 17.

FAIR principles foster efficient work for both humans and machines. Seamless
automation and effective machine learning rely on machine readability enabled
by FAIR data standards18. On a global scale, the collective output of Level 3+
research contributes to a semantic web of datasets and methods, enabling
further aggregation of knowledge.

  3. 3.

FAIR Workflows: Level 3 teams elevate the application of FAIR principles
beyond data management to encompass the entirety of computational workflows,
tracking all data transformations from the raw acquired data through
processing and analysis to the figures in a paper19, 20. These practices
manage the associated code versions, dependencies, environment configurations,
and parameters. Data outputs contain their provenance information describing
their lineage from the original inputs through all computational
transformations. This comprehensive approach ensures that computational
analyses are repeatable and shareable, all while minimizing the potential for
human error.

Formal workflows incorporate best practices for testing code logic, including
unit, regression, and integration testing, often utilizing benchmark datasets.
This rigorous testing framework promotes the reliability and trustworthiness
of research results.

The assimilation of formal workflow specifications has been disparate across
scientific domains. Geosciences and bioinformatics have been at the forefront,
rapidly adapting and benefiting from these advancements. Neuroscience,
especially outside human neuroimaging, has been more reticent, perhaps because
of its vast array of data modalities and the inherent challenges in achieving
standardization21.

Level 3 teams, exemplified by the International Brain Lab and participants in
the NIH U19 program, demonstrate how open science practices significantly
enhance operational capacity22. They view open science not as a burden but as
an opportunity to become more efficient, credible, and accessible in their
research endeavors. This level represents a significant step towards a more
integrated and collaborative scientific community, where shared knowledge and
resources propel research to new heights.

###  2.4 Level 4: Scalable — Introducing SciOps

Levels 4 and 5 transform research operations by adopting technologies for
automation, scalability, and efficient collaboration. We collectively refer to
these methodologies as SciOps, aligning with the principles of other
successful “Ops” disciplines in the technology industry: DevOps, DataOps, and
MLOps.

In the software industry, DevOps cultivates a seamless, semi-automated
pipeline for software development and operations23, 24, 25. DevOps relies on a
set of enabling technologies: containerization, version control,
infrastructure as code (IaC), continuous integration / continuous deployment
(CI/CD), logging and monitoring. Repetitive tasks are automated, transitions
are minimized, computational resources are meticulously orchestrated, and
processes are continuously monitored and rendered observable. The core driver
of productivity in DevOps is the integration of collaborating teams’
activities into a unified automated workflow. Rather than delivering their
outputs separately, these teams contribute to a joint automated pipeline,
guided by well-defined processes. Elimination of hand-offs between teams
reduces errors and shortens development cycles.

Similarly, the field of data analytics has scaled up operations by extending
DevOps into the methodologies of DataOps26, 27, 28 and MLOps29. These
strategies streamline teamwork in data gathering, processing, and analysis and
accelerate the development and deployment of machine learning models. They
encourage teamwork among data specialists through clear communication,
integration, and automation, resulting in more accurate and timely insights.

The emergence of SciOps was surveyed in a 2022 National Academies consensus
study on Automated Research Workflows2, spanning various data-intensive
research domains with one notable exception: neuroscience. These workflows
represent the convergence of computation, lab automation, and artificial
intelligence, spanning across the entire research cycle: experiment design,
observations, simulations, data collection and analysis, and learning from
results to inform further experiments and simulations.

Level 4 leverages technology to dramatically scale and streamline coordinated
activities of multidisciplinary teams collaborating on complex projects. Its
key capabilities include:

  1. 1.

Experiment Automation: Level 4 teams automate experimental processes,
including data collection and analysis, including experiment execution, where
instruments for experiment control, stimuli, sensors, and data acquisition are
integrated into the data pipeline and workflow management.

  2. 2.

DataOps: Level 4 teams implement standard practices for optimizing data
management, processing, and analysis while integrating advanced computational
infrastructure. The primary focus is on achieving efficiency, scalability,
maintaining quality control, and enabling seamless collaboration.

At this level, each project is supported by a meticulously defined shared data
pipeline. This data pipeline serves as a formalized process for aggregating
primary data while ensuring consistency and integrity, acting as the
authoritative source for all downstream activities. As teams progress to
higher maturity levels, the data pipeline extends to encompass all critical
phases of the data lifecycle.

These automated data pipelines exhibit adaptability and transparency, allowing
for the continuous evolution of data analysis methods through collaborative
endeavors. Workflow logic undergoes ongoing validation through automated
testing to minimize the risk of human errors. Software and system design are
structured to efficiently handle large data volumes, guaranteeing fast data
access and movement. Achieving elastic scalability often involves leveraging
cloud and high-performance computing infrastructures, in conjunction with
established best practices for software development and operation.

  3. 3.

Collaboration Environments: Level 4 teams establish streamlined collaboration
environments that enable diverse, distributed teams to access data exploration
capabilities. This includes the use of web-based environments for exploratory
analysis and knowledge exchange, user-friendly interfaces for data
import/export, and the use of community data archives and software
repositories30.

  4. 4.

Teamflow: Level 4 teams establish efficient project management and
communication strategies to support large-scale, multidisciplinary teams.
Project management frameworks are adopted to lower the barriers to
participation. They prioritize project continuity and fair credit tracking for
individual contributions. These teams foster open, transparent, and efficient
communication while ensuring data consistency and integrity. Leadership and
mentorship activities promote flexibility and adaptability. Integration of
rapid software development practices, efficient data management techniques,
and collaborative tools enable teams to work effectively and cohesively within
a complex and scalable research environment.

Only a select few neuroscience research teams approach Level 4 maturity as the
requisite innovations are currently in development. Among those that have made
progress in this direction are well-funded institutes and multi-institution
consortia, including the Allen Institute for Neural Dynamics, e11 Bio, major
BRAIN Initiative initiatives like BICCN 31 and MICrONS 32, as well as the
International Brain Laboratory 33. The Virtual Brain Project, another example,
enables Level 4 operations through open-source containerized cloud services
for multiscale brain simulation and magnetic resonance image processing to
enable scalable and reproducible research34. Open cloud platforms that support
automated data processing and management, such as brainlife.io13, can provide
open and secure environments for researchers to collaborate and implement
reproducible data analysis. The platforms Virtual Research Environment35 and
EBRAINS36 with its federated Health Data Cloud37 peer-to-peer networks38 also
serve as collaborative research environments while protecting data privacy in
building human digital twins for medical research. These entities are at the
forefront of neuroscience data scale and collaboration towards shared research
goals and Level 4 operations. A key challenge for the field is to create an
ecosystem of tools, standards, and platforms to enable diverse research teams
of all sizes and funding levels to adopt these scalable approaches.

###  2.5 Level 5: Optimizing

The ultimate aim of SciOps is the seamless integration of artificial
intelligence (AI) with human cognition, creating a dynamic partnership that
continually refines experiment design and enhances knowledge synthesis through
concurrent computational modeling. Level 5 is distinguished by the strategic
application of advanced technologies and AI to establish a “closed discovery
loop.” This entails the formalization and automation of the process of using
insights derived from data analysis to actively guide subsequent decisions,
ultimately shaping the direction of scientific studies.

Key components establishing AI-accelerated closed-loop studies include:

  1. 1.

Adaptable experiments: Experiment designs are highly configurable, allowing
rapid adaptation to new scientific questions. Parameterized controls include
sensory and optogenetic stimulation, recording protocols, and cognitive tasks.

  2. 2.

Machine learning in the loop: Closed-loop experiments create a continuous
automated feedback loop where machine learning algorithms optimize experiment
controls to maximize knowledge gain about brain structure and function. This
may involve a “digital twin” paradigm, where a simulation of the brain is
continuously refined to match recorded data, facilitating concurrent _in
silico_ experiments. These closed loops can span multiple temporal scales:
from real-time control of experiments to long-term adaptations exploring the
hypothesis space, effectively guiding the scientific study.

Machine learning models powering closed-loop discovery will undergo rapid
development, requiring adopting industry-proven MLOps practices whereby new
models are tested and deployed continuously.

Integrated AI systems observe trends in data, identify new phenomena, propose
hypotheses and new experiments. Future generative AI systems will consider
various information sources, including current literature, existing open
datasets, and ongoing discourse between researchers, embedding this knowledge
into the data pipeline and providing decision support for further inquiry1.

  3. 3.

Human in the loop: While closed-loop experiments automate many processes,
human input and insight remain integral. Level 5 workflows feature a symbiotic
relationship between AI tools and human ingenuity. This hybrid approach
enables rapid hypothesis generation, adaptation of experimental procedures,
effective quality control, and the rapid analysis of large-scale data
collections. To support human participation, the data pipeline and experiment
controls are made observable and explainable.

While no research team in neuroscience (or perhaps in any other discipline)
has fully realized Level 5 operations, existing closed-loop experiments
showcase elements of this vision through the application of machine learning
to optimize experimental conditions based on functional recordings39. These
projects involve the analysis of extensive neurophysiology data in large-scale
collaborations to train deep neural networks for stimulus generation in a
closed-loop manner.

Level 5 of the Neuroscience Operations Maturity Model represents the forefront
of neuroscience research, where AI takes center stage, driving innovation and
enabling researchers to tackle complex questions with unprecedented efficiency
and precision. It will signify a transformative leap in the field’s
capabilities and potential for groundbreaking discoveries.

##  3 What We Need to Advance Modern Scientific Operations in Neuroscience

The evolution of scientific practices, coupled with rapid technological
advancements, demands a strategic shift in how we approach research,
especially in the context of experimental neuroscience. The transformational
power of SciOps methodologies has the potential to reshape the way we think
about scientific endeavors. However, to tap into this potential, we must focus
on specific areas of development and innovation:

###  3.1 Action 1: Adopt this Capability Maturity Model

Community Governance: We invite the community to embrace, enhance, and
collaboratively govern this Capability Maturity Model. This version,
designated as Neuro SciOps CMM v1.0, serves as a starting point, and we are
committed to establishing a roadmap and guidelines for contributions, working
in coordination with the International Neuroinformatics Coordinating Facility
(INCF)40.

A centralized community resource will be established through the INCF and
other platforms to provide access to the maturity model and related resources.
This resource will serve as a valuable reference point for individuals and
organizations interested in advancing scientific operations. To facilitate
community engagement, we envision the creation of a dedicated SciOps resource
and a working group to administer and support the framework and its
collaborative development.

Assessment and Roadmaps: The model will support assessment and certification
processes, enabling organizations to prepare their processes for various
projects and programs. It will also serve as a valuable tool for charting
organizational improvements and technological roadmaps.

By embracing this Capability Maturity Model and actively participating in its
development and application, the scientific community can collectively advance
the field of scientific operations, foster innovation, and drive
transformative progress in research methodologies and practices.

Socialize expectations across the neuroscience community: For this model to be
adopted, the expectations around how neuroscience needs to be performed to
meet 21st century goals will have to be socialized across the neuroscience
community, through integration with training programs, townhall meetings and
workshops at major scientific gatherings.

###  3.2 Action 2: Establish SciOps Methodologies

Current standards and policies in neuroscience data are focused on
standardization and public data sharing, marking a progression towards Level 3
maturity within our model. Achieving Levels 4 and 5 will require new tools,
“SciOps methodologies”, which adapt DevOps principles to the context of
neuroscience experiments. Many practices can be transferred from collaborative
scientific workflow management systems that have gained prominence in
bioinformatics21, 41, 42, 43 and from industry DevOps and DataOps open-source
frameworks and commercial platforms44.

Experiment automation with DevOps, DataOps, and MLOps: Workflow management
technologies should seamlessly integrate with neuroinformatics tools and
methods, setting the stage for more scalable operations as defined in Levels 4
and 5 of our model. The evolution of neuroinformatics tools should prioritize
continuous integration and deployment of research software—whether commercial
or community-driven— encompassing experiment control, data acquisition, and
analysis. Data should be made available in formats and on infrastructure that
allow for scalable storage, processing, and sharing. With the constant flux
and scaling of data, formats and infrastructure evolve rapidly, making it
critical to establish lasting organizational principles to ensure project
continuity. Furthermore, experimental workflows should integrate formal
frameworks for embedding artificial intelligence into the discovery loop.

The creation and widespread adoption of innovative tools tailored specifically
for SciOps have the potential to simplify intricate processes, streamline
tasks, and significantly enhance research efficiency. By embracing this
approach, we can usher in a transformative era of scientific operations
characterized by heightened methodological prowess and overall efficacy.

###  3.3 Action 3: Focus on cyberinfrastructure and digital platforms

Digital platforms have the potential to elevate research teams to Level 3+
maturity without the need for extensive investments in engineering expertise
and custom solutions by individual projects45. However, in the academic
landscape, digital platforms often suffer from limited usability, creating the
perception that shared infrastructure can impede daily activities, presenting
a barrier for sustained adoption. Consequently, some teams opt for building
their own solutions rather than embracing centralized platforms endorsed by
their communities. Unfortunately, these custom-built infrastructures
frequently fall short in terms of service quality, performance, customer
support, and long-term sustainability.

A new generation of academic neuroinformatics platforms, including BrainLife
13, DABI 46, OpenNeuro 15, SPARC 47, and DANDI 11, are positioned to replicate
the success of well-established platforms in the fields of bioinformatics and
biomedical research such as the Galaxy platform, for example48. To do so, they
must evolve beyond data sharing (Level 3) to the automation and scalability of
workflows (Level 4+).

On the other hand, commercial platforms, driven by competitive pressures, are
better positioned to deliver usability, robust customer support, and service
continuity. In the field of neuroscience, emerging commercial platforms such
as DataJoint Works, Inscopix IDEAS, and CodeOcean provide support for
integrated research workflows.

To ensure the success of these platforms, it is essential that they integrate
seamlessly with FAIR data and FAIR workflows, emphasizing transparency,
accessibility, and interoperability of data and computational services. The
widespread adoption of this Capability Maturity Model can assist commercial
technology providers in aligning their business models with the overarching
objective of expanding research capabilities and enhancing the user
experience. Given the diverse nature of the field, it is unlikely that a
single platform or toolset will dominate. Consequently, diverse groups with
common goals and roadmaps can form alliances around shared standards and open-
source frameworks, promoting interoperability, transparency, and
reproducibility across their respective platforms.

To realize this multifaceted vision, it is essential to establish a
comprehensive strategy that encompasses a blend of academic projects,
commercial technology initiatives, and consortial activities, all in alignment
with funding policies. This multifaceted approach will guide the formulation
of research projects, the marketing efforts of commercial technology
providers, and the policies of funding agencies, ultimately fostering the
creation of a unified and sustainable ecosystem.

## Acknowledgements

We would like to thank the many community members who contributed to the
discussions and reviews of this manuscript and raised key community issues,
including David Feng, Andreas S. Tolias, Marisel Villafañe-Delgado, Lindsey
Kitchell, Daniel Xenes, and others. DY, BW, TN, MK, KG, and ECJ were supported
in part by the NIH (award R44 NS129492). YOH was supported in part by the NIH
(awards 1 R24 MH117295 and 2 P41 EB019936-06A1 R). The content is solely the
responsibility of the authors and does not necessarily represent the official
views of the National Institutes of Health or other supporting institutions.

## References

  * 1 Wang, H. _et al._ Scientific discovery in the age of artificial intelligence.  _Nature_ 620, 47–60 (2023).
  * 2 NASEM.  _Automated Research Workflows for Accelerated Discovery: Closing the Knowledge Discovery Loop_ (National Academies of Engineering and Medicine, 2022).  URL https://nap.nationalacademies.org/catalog/26532/automated-research-workflows-for-accelerated-discovery-closing-the-knowledge-discovery.
  * 3 Paulk, M. C.  A history of the capability maturity model for software.  _ASQ Software Quality Professional_ 12, 5–19 (2009).
  * 4 Chrissis, M. B., Konrad, M. & Shrum, S.  _CMMI for development: guidelines for process integration and product improvement_ (Pearson Education, 2011).
  * 5 Bush, K. A., Calvert, M. L. & Kilts, C. D.  Lessons learned: A neuroimaging research center’s transition to open and reproducible science.  _Frontiers in big Data_ 82 (2022).
  * 6 Feyerabend, P.  _Against method: Outline of an anarchistic theory of knowledge_ (Verso Books, 2020).
  * 7 Artaza, H. _et al._ Top 10 metrics for life science software good practices.  _F1000Research_ 5 (2016).
  * 8 Eglen, S. J. _et al._ Toward standard practices for sharing computer code and programs in neuroscience.  _Nature neuroscience_ 20, 770–773 (2017).
  * 9 Wilkinson, M. D. _et al._ The FAIR guiding principles for scientific data management and stewardship.  _Scientific data_ 3, 1–9 (2016).
  * 10 Gorgolewski, K. J. _et al._ The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments.  _Scientific data_ 3, 1–9 (2016).
  * 11 Rübel, O. _et al._ The neurodata without borders ecosystem for neurophysiological data science.  _Elife_ 11, e78362 (2022).
  * 12 Subash, P. _et al._ A comparison of neuroelectrophysiology databases.  _Scientific Data_ 10, 719 (2023).
  * 13 Hayashi, S. _et al._ brainlife.io: A decentralized and open source cloud platform to support neuroscience research.  _ArXiv_ (2023).
  * 14 Hider Jr, R. _et al._ The brain observatory storage service and database (BossDB): a cloud-native approach for petascale neuroscience discovery.  _Frontiers in Neuroinformatics_ 16, 828787 (2022).
  * 15 Markiewicz, C. J. _et al._ The OpenNeuro resource for sharing of neuroscience data.  _Elife_ 10, e71774 (2021).
  * 16 Halchenko, Y. _et al._ Datalad: distributed system for joint management of code, data, and their relationship.  _Journal of Open Source Software_ 6 (2021).
  * 17 Kalantari, A. _et al._ How to establish and maintain a multimodal animal research dataset using datalad.  _Scientific data_ 10, 357 (2023).
  * 18 Huerta, E. _et al._ FAIR for AI: An interdisciplinary and international community building perspective.  _Scientific Data_ 10, 487 (2023).
  * 19 Goble, C. _et al._ FAIR computational workflows.  _Data Intelligence_ 2, 108–121 (2020).
  * 20 Deelman, E. _et al._ The future of scientific workflows.  _The International Journal of High Performance Computing Applications_ 32, 159–175 (2018).
  * 21 Wratten, L., Wilm, A. & Göke, J.  Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers.  _Nature methods_ 18, 1161–1168 (2021).
  * 22 Bonacchi, N. _et al._ A modular architecture for organizing, processing and sharing neurophysiology data.  _Nature Methods_ 20, 403–407 (2023).
  * 23 Ebert, C., Gallardo, G., Hernantes, J. & Serrano, N.  DevOps.  _IEEE Software_ 33, 94–100 (2016).
  * 24 Leite, L., Rocha, C., Kon, F., Milojicic, D. & Meirelles, P.  A survey of DevOps concepts and challenges.  _ACM Computing Surveys (CSUR)_ 52, 1–35 (2019).
  * 25 Teixeira, D. _et al._ A maturity model for DevOps.  _International Journal of Agile Systems and Management_ 13, 464–511 (2020).
  * 26 Gartner.  Gartner Hype Cycle for Data Management Positions Three Technologies in the Innovation Trigger Phase in 2018.  https://www.gartner.com/en/newsroom/press-releases/2018-09-11-gartner-hype-cycle-for-data-management (2018).  [Online; accessed 22-Dec-2023].
  * 27 Rodriguez, M., de Araújo, L. J. P. & Mazzara, M.  Good practices for the adoption of DataOps in the software industry.  In _Journal of Physics: Conference Series_ , vol. 1694, 012032 (IOP Publishing, 2020).
  * 28 Atwal, H.  _Practical DataOps: Delivering agile data science at scale_ (Springer, 2019).
  * 29 Mäkinen, S., Skogström, H., Laaksonen, E. & Mikkonen, T.  Who needs MLOps: What data scientists seek to accomplish and how can MLOps help?  In _2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN)_ , 109–112 (IEEE, 2021).
  * 30 Milewicz, R. _et al._ DevOps pragmatic practices and potential perils in scientific software development.  In _International Congress on Information and Communication Technology_ , 629–647 (Springer, 2023).
  * 31 Network, B. I. C. C.  A multimodal cell census and atlas of the mammalian primary motor cortex.  _Nature_ 598, 86–102 (2021).
  * 32 Consortium, M. _et al._ Functional connectomics spanning multiple areas of mouse visual cortex.  _BioRxiv_ 2021–07 (2021).
  * 33 Abbott, L. F. _et al._ An international laboratory for systems and computational neuroscience.  _Neuron_ 96, 1213–1218 (2017).
  * 34 Schirner, M. _et al._ Brain simulation as a cloud service: The virtual brain on EBRAINS.  _NeuroImage_ 251, 118973 (2022).
  * 35 VRE.  Virtual Research Environment.  https://vre.charite.de/vre/ (2023).  [Online; accessed 22-Dec-2023].
  * 36 EBRAINS.  EBRAINS.  https://www.ebrains.eu/ (2023).  [Online; accessed 22-Dec-2023].
  * 37 HealthDataCloud.  HealthDataCloud.  https://www.healthdatacloud.eu/ (2023).  [Online; accessed 22-Dec-2023].
  * 38 eBRAIN Health.  eBRAIN-Health.  https://ebrain-health.eu/home.html (2023).  [Online; accessed 22-Dec-2023].
  * 39 Walker, E. Y. _et al._ Inception loops discover what excites neurons most using deep predictive models.  _Nature neuroscience_ 22, 2060–2065 (2019).
  * 40 Abrams, M. B. _et al._ A standards organization for open and FAIR neuroscience: the international neuroinformatics coordinating facility.  _Neuroinformatics_ 20, 25–36 (2022).
  * 41 Di Tommaso, P. _et al._ Nextflow enables reproducible computational workflows.  _Nature biotechnology_ 35, 316–319 (2017).
  * 42 Köster, J. & Rahmann, S.  Snakemake—a scalable bioinformatics workflow engine.  _Bioinformatics_ 28, 2520–2522 (2012).
  * 43 Vivian, J. _et al._ Toil enables reproducible, open source, big biomedical data analyses.  _Nature biotechnology_ 35, 314–316 (2017).
  * 44 Bhat, M. _et al._ Magic quadrant for devops platforms.  https://www.gartner.com/doc/reprints?id=1-2DW4I0FF&ct=230601&st=sb (2023).  [Online; accessed 22-Dec-2023].
  * 45 Sandström, M. _et al._ Recommendations for repositories and scientific gateways from a neuroscience perspective.  _Scientific Data_ 9, 212 (2022).
  * 46 Duncan, D. _et al._ Data archive for the brain initiative (DABI).  _Scientific Data_ 10, 83 (2023).
  * 47 Bandrowski, A. _et al._ SPARC data structure: Rationale and design of a fair standard for biomedical research data.  _bioRxiv_ 2021–02 (2021).
  * 48 Afgan, E. _et al._ The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update.  _Nucleic acids research_ 50 (2022).


# MISFIT-V: Misaligned Image Synthesis and Fusion using Information from
Thermal and Visual

Aadhar Chauhan
University of Washington
<EMAIL_ADDRESS>Isaac Remy
University of Washington
<EMAIL_ADDRESS>Danny Broyles
University of Washington
<EMAIL_ADDRESS>Karen Leung
University of Washington
<EMAIL_ADDRESS>
###### Abstract

Detecting humans from airborne visual and thermal imagery is a fundamental
challenge for Wilderness Search-and-Rescue (WiSAR) teams, who must perform
this function accurately in the face of immense pressure. The ability to fuse
these two sensor modalities can potentially reduce the cognitive load on human
operators and/or improve the effectiveness of computer vision object detection
models. However, the fusion task is particularly challenging in the context of
WiSAR due to hardware limitations and extreme environmental factors. This work
presents Misaligned Image Synthesis and Fusion using Information from Thermal
and Visual (MISFIT-V), a novel two-pronged unsupervised deep learning approach
that utilizes a Generative Adversarial Network (GAN) and a cross-attention
mechanism to capture the most relevant features from each modality.
Experimental results show MISFIT-V offers enhanced robustness against
misalignment and poor lighting/thermal environmental conditions compared to
existing visual-thermal image fusion methods. The code is available at
GitHub.111https://github.com/Aadharc/Visual_Thermal_Image_Fusion.git

##  1 Introduction

Search and rescue teams worldwide have increasingly relied on uncrewed aerial
vehicles (UAVs) equipped with visual and thermal imaging sensors to enhance
the rescuer’s ability to detect and locate lost or injured persons. Often,
these operations take place in wilderness environments (i.e., wilderness
search and rescue [WiSAR]) featuring hazardous and difficult-to-access
terrain, and in various weather and lighting conditions which affect the
quality of information obtained from visual and thermal modalities.
Ultimately, these conditions affect the ability to detect the presence or
absence of missing persons in the imagery. The detection task in a search and
rescue mission, whether performed by human imagery analysts, UAV operators, or
computer vision algorithms, is a fundamentally crucial function which can make
the difference between mission success or failure.

Thus, developing algorithms that can detect humans more reliably and in
adverse lighting and weather conditions is of utmost importance, and
multimodal image fusion is an active area of research that offers many
advantages for this application.

Figure 1: System Conceptual Overview. Images from visual (RGB) and thermal
(IR) cameras capturing the same scene are misaligned due to physical
properties. MISFIT-V resolves this misalignment while emphasizing useful
features from both modalities in the fused output.

Multi-modal image fusion is the act of combining information from multiple
image modalities into a useful representation that can be more easily used for
human or robotic perception-related tasks. This remains an important area of
research with applications in medical diagnosis (fusing CT and MRI scans [1]),
remote sensing (detecting environmental anomalies from satellite imagery [19,
21]), and application featured in this paper, which is visual-thermal image
fusion for enhanced human detection in UAV-aided search and rescue missions.
However, achieving accurate and effective fusion between visual and thermal
image modalities is particularly challenging in practice due to a variety of
factors, including differences in sensor resolution, noise characteristics,
and spatial and temporal misalignment which affect the image registration
process.

In this paper, we address the problem of fusing misaligned visual and thermal
image pairs using unsupervised deep learning methods and provide qualitative
and quantitative analysis of the resulting fused image quality. The proposed
method, Misaligned Image Synthesis and Fusion using Information from Thermal
and Visual (MISFIT-V) accounts for practical considerations of WiSAR
operations, and presents the following advantages:

  * •

Image registration is not required

  * •

Ground truth fused images are not needed during model training

  * •

The fused output is human-interpretable

  * •

The fused output balances visual/thermal features

###  1.1 Problem Statement

The goal is to fuse a visual-thermal image pair
$(I_{\mathrm{RGB}},I_{\mathrm{IR}})\in\mathbb{R}^{H\times W\times
3}\times\mathbb{R}^{H\times W\times 1}$222To ensure compatibility with the
network architecture, the images undergo a preprocessing step that adjusts
their sizes to be the same before feeding them into the network., no
necessarily aligned, into a single image
$I_{\mathrm{fus}}\in\mathbb{R}^{H\times W\times 3}$ whereby features are
aligned, and salient features from each image are combined. Here, $H$ & $W$
represent the height and width of the corresponding images, and 3 & 1 are the
number of channels in respective images. Motivated by the practical
considerations of WiSAR operations, we further make the following assumptions
in our problem setup:

The image pairs are not aligned: Given visual and thermal sensors function
with different optical characteristics (e.g., field of view, resolution, and
lens distortions), resulting in sets of misaligned visual and thermal images
and largely uncorrelated feature sets. This makes traditional image
registration/fusion techniques inadequate.

Ground truth fused images do not exist: Indeed obtaining fused images is the
challenge, and manual image fusion is not a viable solution to obtain ground
truth images.

Fused image must be human-interpretable: While the end goal is to use the
fused image to aid in human detection, the fused image must remain human-
interpretable given the human-on-the-loop nature of WiSAR missions, i.e.,
human operators monitor the video feed while a human detection algorithm runs
concurrently.

###  1.2 Organization

The rest of this paper is organized as follows. Section LABEL:sec:related_work
discusses the relevant background and related work in image fusion. Section 3
describes the architecture of MISFIT-V, and Sections 4 and 5 report the
experimental results and conclusions, respectively.

##  2 Related Work

A vast number of approaches to the image fusion problem have been explored in
the literature (see [12] for a review). We present a brief review on deep-
learning-based fusion algorithms which have demonstrated state-of-the-art
performance in this area.

###  2.1 Visual & Thermal Image Fusion

Many standard image fusion methods for thermal and visual images, such as
those proposed in [2, 7, 11, 15], rely on the input images that are aligned at
the pixel level. This step of visual-thermal image alignment often requires
precise hardware-level calibration and image processing, and misaligned
features are the primary sources of error for these fusion algorithms. In many
practical scenarios where visual and thermal cameras are used, the inherent
differences between the two sensors (resolution, field of view, lens effects,
and noise characteristics) make these images misaligned.

Moreover, visual and thermal imaging sensors use distinctly different
operating principles (visible spectrum light versus infrared radiation), often
resulting in very little correspondence between each modality’s features [14].

###  2.2 Multimodal Image Fusion with Generative Adversarial Networks (GANs)

More recently, several methods ([9, 20, 22]) propose using image-to-image
translation as a way to address the lack of common features between visual and
thermal images. Image-to-image translation is the act of taking images from
one domain and transforming them so they have the style (or characteristics)
of images from another domain. These methods [18] use the Generative
Adversarial Network (GAN) architecture as the backbone to translate the image
from one modality to the other. However, the dataset [16] used for training
the GAN architecture in these works is pre-registered and aligned at the pixel
level, which can cause problems with respect to scalability. Since these
methods require pixel-aligned multi-modal images, they cannot be utilized in
many real-world applications where the alignment between visual and thermal
sensor pairs is unknown a priori. For instance, a recent dataset, called
WiSARD [4], features visual and thermal image pairs taken from a UAV’s
perspective with annotations for human detection in a wilderness environment;
however, the images are not perfectly aligned with each other on the pixel
level.

###  2.3 Cross-Attention Mechanism

Natural language processing (NLP) research made substantial advancements in
recognizing the relationships between various input sequence segments with the
introduction of the Transformer model and the attention mechanism, and in
particular the cross-attention mechanism is commonly used in deep learning
models for tasks involving multiple modalities [17]. Numerous studies have
used a transformer model to successfully perform multi-modal data processing
tasks, using a cross-attention module that enables the model to learn what
information from one modality is relevant when analyzing features from another
modality. For instance, [6] combined multi-scale picture characteristics using
cross-attention, while [3] combined visual and LiDAR image features using
self-attention and cross-attention.

##  3 Methodology

In this section, we described the proposed neural architecture of MISFIT-V,
and motivate the loss function used to train the model.

###  3.1 Proposed Architecture

Figure 2 depicts our proposed model, which was inspired by the idea of using
two discriminators presented in [10, 13], leverages a Generative Adversarial
Network (GAN) architecture as its backbone. The inclusion of two
discriminators enables the preservation of information from both input
modalities, ensuring that the distinctive features and characteristics of each
modality are effectively retained. Next, we describe each component of the GAN
architecture.

Figure 2: MISFIT-V Training Pipeline. Thermal (IR) and visual (RGB) images are
fed into a generator (orange block) consisting of a cross-attention module and
CNN to produce a fused image. The fused image is fed into both discriminator
networks, encouraging a balanced set of features from both images.

####  3.1.1 Generator with Cross Attention

The generator, shown in Figure 3, is designed to produce a fused image given a
visual-thermal image pair. The generator network is comprised of two separate
Convolutional Neural Networks (CNNs) for downsampling and feature extraction
from the input images. Additionally, a Cross-Attention Network [5, 17] is
incorporated to capture meaningful and unique features from each modality,
i.e., “best of both worlds”, considering the diverse aspects that thermal and
visible images focus on in the same scene (see Appendix A). The utilization of
cross-attention eliminates the need for explicit alignment of the images,
thereby addressing the fusion of misaligned input images. The outputs of the
cross-attention network are a tuple of cross-attention maps with respect to
both modalities, which are multiplied with the downsampled features and fed
into an upsampling CNN. Finally, the outputs from upsampling CNNs are
concatenated and fed into the U-Net which generates the fused image by
integrating the information from both modalities to produce a more
comprehensive and meaningful fused image (see Figure 3).

####  3.1.2 Dual Discriminators

Given that no ground truth data exists, the proposed discriminator module is
comprised of two discriminator networks; one for visual and another for
thermal, see Figure 2. Each discriminator takes a concatenated image comprised
of the original (visual or thermal) and fused images and classifies them as
either real or fake. In this way, the generator is discouraged from passing
only the features from one modality through, so that it ultimately achieves a
balanced set of features from each modality in the fused output.

Figure 3: MISFIT-V Generator Architecture. Input images are fed into a
downsampling CNN (‘Down’) separately to retrieve their features. These
features are then fed into the cross-attention network to calculate the cross-
attention map between the modalities, which are multiplied with the
downsampled features and concatenated to form the input to the U-Net CNN,
which generates a fused image.

###  3.2 Loss Function

Here, we describe the loss function used to train MISFIT-V. We employ an
adversarial loss function to train the discriminators and generator in order
to generate high-quality fused images. The adversarial loss for the
discriminator, which corresponds to a specific modality X (either thermal/IR
or visual/RGB), is defined as follows:

$\mathcal{L}_{\text{adv,X}}=-\log
D_{\text{X}}(I_{\text{X}})-\log(1-D_{\text{X}}(I_{\text{fus}})),$ (1)

where $D_{\text{X}}(I_{\text{X}})$ represents the probability that
$I_{\text{X}}$ is classified as modality X , and
$D_{\text{X}}(I_{\text{fus}})$ represents the probability that
$I_{\text{fus}}$ is classified as modality X by discriminator.

The generator loss is defined as the sum of the adversarial losses from both
discriminators, weighted by hyperparameters $\lambda_{\text{IR}}$ and
$\lambda_{\text{RGB}}$, respectively:

$\mathcal{L}_{\text{gen}}=\lambda_{\text{IR}}\cdot\mathcal{L}_{\text{adv,IR}}+\lambda_{\text{RGB}}\cdot\mathcal{L}_{\text{adv,RGB}}$
(2)

where $\lambda_{\text{IR}}$ and $\lambda_{\text{RGB}}$ control the relative
importance of the respective losses in the overall generator loss. Here, the
adversarial losses encourage the generator to create fused images that are
indistinguishable from thermal images and visual images by training against
two discriminators that try to classify them.

In addition to the adversarial losses, a Kullback-Leibler (KL) Divergence loss
is used to compare the fused image generated by the generator with the
original visual and thermal images in terms of their distribution. The KL
Divergence loss is defined as:

$\mathcal{L}_{\text{KL}}=\text{KL}(I_{\text{fus}}||I_{\text{IR}})+\text{KL}(I_{\text{fus}}||I_{\text{RGB}})$
(3)

where $\text{KL}(I_{\text{fus}}||I_{\text{IR}})$ and
$\text{KL}(I_{\text{fus}}||I_{\text{RGB}})$ represent the KL Divergence
between the fused image $I_{\text{fus}}$ and the thermal image
$I_{\text{IR}}$, and between the fused image $I_{\text{fus}}$ and the visual
image $I_{\text{RGB}}$, respectively. Furthermore, an L1 loss is utilized to
calculate the pixel-wise differences between the fused image and each of the
original images. This loss can be expressed as:

$\mathcal{L}_{\text{L1}}=\|I_{\text{fus}}-I_{\text{IR}}\|_{1}+\|I_{\text{fus}}-I_{\text{RGB}}\|_{1},$
(4)

where $\|\cdot\|_{1}$ denotes the L1 norm. The overall loss,

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{gen}}+\lambda_{\text{KL}}\mathcal{L}_{\text{KL}}+\lambda_{\text{L1}}\mathcal{L}_{\text{L1}}$
(5)

is the sum of the generator loss, the KL divergence loss, and L1 loss weighted
by hyperparameters $\lambda_{\text{KL}}$, and $\lambda_{\text{L1}}$ that
control the relative importance of the KL divergence loss and L1 loss.

##  4 Experimental Results

###  4.1 Dataset and Training Details

The model was trained using 2752 pairs of thermal and visual images from the
WiSARD dataset [4], with an 80:20 split for training and validation, and a
separate test dataset of 200 pair of images was employed to evaluate the
performance of the trained model. The network is trained for 20 epochs, using
a learning rate of $1\times 10^{-4}$. The hyperparameters for the training
process are set as follows: $\lambda_{\text{KL}}=10$,
$\lambda_{\text{L1}}=100$, $\lambda_{\text{IR}}=1$, and
$\lambda_{\text{RGB}}=1$.

###  4.2 Qualitative Analysis

We demonstrate that the fused images provide a clearer representation of the
environment than both modalities alone. From Figure 4, we see that our method
produces well-fused images that retain the terrain features but extract the
bright human silhouettes from the thermal image. However, our method still has
limitations; for example, when objects of interest are small in size, the
attention mechanism may encounter challenges in accurately determining the
essential features from both modalities, leading to the emergence of ghost
artifacts in the fused image (see the fourth row in Figure 4).

Figure 4: MISFIT-V Results. Each row, from left to right, shows a scene’s
thermal representation, its visual representation, and finally the resulting
fused image via MISFIT-V, in a wilderness environment. The yellow bounding
boxes highlight the locations of humans.

###  4.3 Quantitative Comparison

To quantify the fusion results, we evaluate the extent to which the
information from each modality is preserved. This analysis involves
calculating particular metrics individually for thermal and visual images
against the fused image, to measure the level of information retention in the
fusion process. We compare MISFIT-V against SeAFusion [15], another method
that exhibits state-of-the-art performance for visual-thermal image fusion.
Given the formulation of SeAFusion, we had to evaluate both methods on an
autonomous driving dataset [8] which contains more structure than WiSAR
settings and had ground truth labels. We compare against five metrics: Mean-
squared-error (MSE), universal quality index (UQI), multi-scale structural
similarity (MSSSIM), normalized mutual information (NMI), and the peak signal-
to-noise ratio (PSNR). The y-axis of the comparison (see Figure 5) represents
the numerical values corresponding to each metric. For brevity, we have
presented the results for three metrics in this section which have been
normalized. The plots for the remaining metrics can be found in Appendix B.
One interesting trend in our results is that MISFIT-V prioritizes information
from the thermal modality to a greater extent and performs only marginally
worse in retaining visual information when compared to SeAFusion. While
MISFIT-V outperforms SeAFusion in some aspects and only slightly worse in
others, it offers a significant advantage over SeAFusion by eliminating the
need for semantic labeling and ground truth data, thus enhancing its
scalability across diverse datasets. This characteristic enhances the
applicability and adaptability of MISFIT-V in a wider range of scenarios.

Figure 5: Method Comparison. This plot shows the results of three performance
metrics comparing thermal and visual images against fused images generated by
MISFIT-V and SeAFusion.

###  4.4 Ablation Study

In the pursuit of refining and optimizing our proposed methodology, we
conducted an ablation study to meticulously analyze the effects of various
modifications on the performance of our model. Through a series of controlled
experiments, we aimed to dissect the contribution of specific components and
choices within the architecture and total loss function. Here, we present the
findings of our ablation study, comparing the original method with distinct
variations: one involving the adjustment of the weightage of certain loss
functions (Figure 6) and excluding the cross attention mechanism between
thermal and visual data.

Figure 6: Comparison of image fusion performance using metrics ‘MSSSIM’. The
first column, labeled as ‘Original’, presents the scores achieved by the
original method. In the second column, labeled as ‘$\lambda_{\mathrm{L1}}=1$’
the fusion performance is shown when the weightage of L1 loss is adjusted to
1. The third column displays the results obtained when the KL loss term is
omitted. Notably, the quality of the fused image is observed to decrease when
the KL loss is omitted, and this degradation is further exacerbated when the
$\lambda_{\mathrm{L1}}$ is set to 1. This visually emphasizes the significance
of the KL loss term and the weightage of L1 loss in maintaining the quality of
the generated fused images.

####  4.4.1 Impact of Loss Function Variations on Fused Image Quality

An essential component of our proposed method revolves around the integration
of L1 loss within the comprehensive loss function, aimed at optimizing the
fusion procedure. To assess the significance of this specific loss term, we
embarked on an experiment by modifying the weightage attributed to the L1
loss. In particular, we reduced the weightage from its original value of 100
to a much lower value of 1. The rationale behind this manipulation was to
ascertain whether diminishing the emphasis on the L1 loss would result in
discernible alterations in the quality of the generated fused images. See
Figure 6 and 12.

The outcomes of this experiment unveiled an intriguing insight. By reducing
the L1 loss weightage, we observed a distinct deterioration in the
comprehensibility of the resultant fused images. In other words, when the
weightage was lowered from 100 to 1, the fused images exhibited a decrease in
their interpretability and coherence. This phenomenon suggests that the L1
loss component indeed plays a pivotal role in shaping the clarity and visual
coherence of the fused images. As such, its significance as a contributing
factor to the overall loss function is highlighted, reinforcing the critical
importance of its weightage within the fusion process.

In another variation, we examined the consequences of omitting the Kullback-
Leibler (KL) loss term while maintaining the weightage of L1 loss at the
original value of 100. This omission aimed to explore the repercussions of
excluding the KL loss term on the final quality of the fused images. The
subsequent analysis, as evidenced by the diverse metric plots presented below,
offers valuable insights into the outcomes of these variations and their
implications on the fused image quality. See Figure 6. For additional
information and graphical representations for other metrics, please refer to
Appendix C in this paper.

The experimental results shed light on an intriguing phenomenon. When the KL
loss was removed from the loss function, we observed a discernible reduction
in the quality of the fused images. This reduction was evident across various
metrics that assess the image quality, underscoring the importance of the KL
loss in enhancing the fusion process. By omitting the KL loss, which serves as
a vital bridge between the latent space and the generated image, the model’s
ability to capture and reproduce intricate visual features was compromised.
Consequently, the fused images exhibited a lower level of fidelity and
coherence.

####  4.4.2 Impact of Attention Mechanism on Fusion Quality

The cross attention mechanism serves as a pivotal bridge between thermal and
visual data, enabling the model to capture distinct yet complementary
information from both modalities. In this ablation variation, we removed the
cross attention mechanism entirely from the architecture to evaluate its
influence on the fusion process.

The comparison is presented in Figure 7, where the right image displays the
fused images generated without the attention mechanism and the left image
showcases the fused images produced with the attention mechanism. Notably, the
fused image generated without attention exhibited ghost artifacts and the
inclusion of visual features, which can be attributed to a lack of emphasis on
the essential characteristics of both modalities during the fusion process. In
contrast, the fused images generated with the attention module demonstrated a
distinct improvement in terms of quality and coherence. The attention
mechanism effectively identifies and prioritizes significant features while
minimizing the impact of less relevant visual features. This results in a more
balanced and comprehensive fused image that accurately represents the salient
information present in both thermal and visual modalities.

Figure 7: A comparison between fused images generated with and without the
attention mechanism is presented. The right side showcases the fused image
generated without attention, where the inclusion of visual features leads to
the emergence of ghost artifacts. On the left side, the fused image produced
with the attention module is displayed, demonstrating that the attention
mechanism effectively omits less important visual features, resulting in a
more coherent and comprehensible image.

The visual comparison is further supported by quantitative assessments, where
metrics such as Mean-Squared Error (MSE), Multi-Spectral Structural Similarity
Index (MSSSIM), Normalised Mutual Information (NMI), Universal Quality Index
(UQI) and Peak Signal-to-Noise Ratio (PSNR) are employed to quantify the
improvement in image quality achieved through the attention mechanism. Please
refer to Figure 8 for a detailed comparison of metrics including UQI, MSSSIM,
and MSE. For additional information and graphical representations for other
metrics, please consult Appendix D in this paper.

Figure 8: The plot illustrates the effect of utilizing the attention mechanism
in the image fusion process. In the columns named ‘With-Attention’, fusion
results obtained with the attention mechanism are displayed, while in the
columns named ‘No-Attention’, results without the attention mechanism are
presented. Evidently, the quality of the fused images noticeably decreases
when the attention mechanism is not employed, underscoring its vital role in
enhancing the fusion process by selectively focusing on significant features
from both modalities.

##  5 Conclusion

We have presented MISFIT-V, a novel approach for visual-thermal image fusion
leveraging a GAN architecture with two discriminators and a cross attention
mechanism to blend crucial information from both modalities and hence
eliminating the need to align the images altogether. The experimental results
demonstrate the robustness and superior performance of MISFIT-V, outperforming
a state-of-the-art baseline method while effectively handling misaligned
images. By enabling a more complete representation of the environment,
MISFIT-V has the potential to enhance WiSAR mission effectiveness and
alleviate the cognitive load on human operators.

## References

  * [1] M. A. Azam, K. B. Khan, S. Salahuddin, E. Rehman, S. A. Khan, M. A. Khan, S. Kadry, and A. H. Gandomi.  A Review on Multimodal Medical Image Fusion: Compendious Analysis of Medical Modalities, Multimodal Databases, Fusion Techniques and Quality Metrics.  Computers in Biology and Medicine, 2022.
  * [2] A. Azarang, H. E. Manoochehri, and N. Kehtarnavaz.  Convolutional Autoencoder-Based Multispectral Image Fusion.  IEEE Access, 2019.
  * [3] R. Bose, S. Pande, and B. Banerjee.  Two Headed Dragons: Multimodal Fusion And Cross Modal Transactions.  In IEEE International Conference on Image Processing, 2021.
  * [4] D.* Broyles, C.* Hayner, and K. Leung.  WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness Search and Rescue.  In IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, 2022.
  * [5] G. Cai, Y. Zhu, Y. Wu, X. Jiang, J. Ye, and D. Yang.  A Multimodal Transformer to Fuse Images and Metadata for Skin Disease Classification.  The Visual Computer, 2022.
  * [6] C. R. Chen, F. Quanfu, and R. Panda.  Crossvit: Cross-Attention Multi-Scale Vision Transformer for Image Classification.  In IEEE Int. Conf. on Computer Vision, 2021.
  * [7] F. Farahnakian and J. Heikkonen.  Deep Learning Based Multi-Modal Fusion Architectures for Maritime Vessel Detection.  Remote Sensing, 2020.
  * [8] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada.  MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes.  In IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, 2017.
  * [9] P. Isola, J. Zhu, T. Zhou, and A. Efros.  Image-to-Image Translation with Conditional Adversarial Networks.  In IEEE Conf. on Computer Vision and Pattern Recognition, 2017.
  * [10] J. Li, H. Huo, C. Li, R. Wang, and Q. Feng.  AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks.  IEEE Transactions on Multimedia, 2021.
  * [11] Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward, and X. Wang.  Deep Learning for Pixel-Level Image Fusion: Recent Advances and Future Prospects.  Information Fusion, 2018.
  * [12] W. Ma, K. Wang, J. Li, S. X. Yang, J. Li, L. Song, and Q. Li.  Infrared and Visible Image Fusion Technology and Application: A Review.  Sensors, 2023.
  * [13] D. Rao, T. Xu, and X. Wu.  TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network.  IEEE Transactions on Image Processing, 2023.
  * [14] P. Ricaurte, C. Chilán, C. A. Aguilera-Carrasco, B. X. Vintimilla, and A. D. Sappa.  Feature Point Descriptors: Infrared and Visible Spectra.  Sensors, 2014.
  * [15] L. Tang, J. Yuan, and J. Ma.  Image Fusion in the Loop of High-Level Vision Tasks: A Semantic-Aware Real-Time Infrared and Visible Image Fusion Network.  Information Fusion, 2022.
  * [16] A. Toet, J. K. IJspeert, A. M. Waxman, and M. Aguilar.  Fusion of Visible and Thermal Imagery Improves Situational Awareness.  Displays, 1997.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  In Conf. on Neural Information Processing Systems, 2017.
  * [18] D. Wang, J. Liu, X. Fan, and R. Liu.  Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration.  Available at https://arxiv.org/abs/2205.11876, 2022.
  * [19] D. Xiu, Z. Pan, Y. Wu, and Y. Hu.  MAGE: Multisource Attention Network With Discriminative Graph and Informative Entities for Classification of Hyperspectral and LiDAR Data.  IEEE Transactions on Geosciences and Remote Sensing, 2022.
  * [20] Q. Yang, N. Li, Z. Zhao, X. Fan, E. I. Chang, Y. Xu, et al.  MRI Cross-Modality Image-to-Image Translation.  Scientific reports, 2020.
  * [21] Y. Yang, W. Wan, S. Huang, F. Yuan, S. Yang, and Y. Que.  Remote Sensing Image Fusion Based on Adaptive IHS and Multiscale Guided Filter.  IEEE Access, 2016.
  * [22] J. Zhu, T. Park, P. Isola, and A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE Int. Conf. on Computer Vision, 2017.

##  Appendix A Cross-Attention

We incorporated cross-attention within our generator network to selectively
focus on significant features from both image modalities. Cross-attention
allows the generator to prioritize relevant information from each modality
during the fusion process, enhancing the quality of the generated output. In
the attention mechanism of the transformer architecture [17], queries, keys,
and values are fundamental components. Queries represent the information that
is sought or being searched for, while keys and values hold the information to
be queried. In the context of cross-attention, queries from one modality are
exchanged with queries from the other modality. We have visual and thermal
modalities and queries from the visual modality are exchanged with queries
from the thermal modality. (See Figure 3). This exchange enables the model to
capture and leverage the complementary information from both modalities. By
attending to important features in the thermal modality using visual queries,
and vice versa, the generator can effectively combine and synthesize the
relevant information from each modality for accurate fusion. See Figure 9.

Figure 9: Cross Attention. The heatmaps on the right illustrate the regions of
the images on the left that receive higher attention, with warmer colors
indicating a greater degree of focus. In particular, areas with a reddish hue
indicate heightened attention and prioritization.

##  Appendix B NMI and PSNR Comparison

We present the results for image fusion using the metrics of Normalized Mutual
Information (NMI) (see Figure 10) and Peak Signal-to-Noise Ratio (PSNR) (see
Figure 11). The plots for NMI and PSNR provide a comprehensive evaluation of
the image fusion performance for both thermal and visual modalities and they
follow a similar trend as shown by the other three metrics in Figure 5.

Figure 10: Normalized Mutual Information Comparison Figure 11: Peak Signal
Noise Ratio Comparison

##  Appendix C Impact of Loss Function Variations on Fused Image Quality

In the appendix section, we delve deeper into our experimentation process,
focusing on critical components of our proposed method. Specifically, we
explore the significance of the L1 loss within our comprehensive loss function
and the implications of omitting the Kullback-Leibler (KL) loss term as
explained in 4.4.1. These insights provide valuable context for our research
and shed light on the crucial factors influencing the quality of fused images.
Please refer to Figure 12, 13 and 14 for visual representations of our
findings in these experiments.

Figure 12: Comparison of image fusion performance using metrics ‘UQI’. Figure
13: Comparison of image fusion performance using metrics ‘PSNR’. Figure 14:
Comparison of image fusion performance using metrics ‘MSE’.

##  Appendix D Impact of Attention Mechanism on Fusion Quality

In the appendix section, we delve deeper into our experimentation process,
focusing on critical components of our proposed method. Specifically, we
explore the significance of the attention mechanism incorporated in the
generator of our GAN, as explained in 4.4.2. These insights provide valuable
context for our research and shed light on the crucial factors influencing the
quality of fused images. Please refer to Figure 15 and 16 for visual
representations of our findings in these experiments.

Figure 15: Comparison of image fusion performance using metrics ‘PSNR’ to
understand the impact of attention mechanism. Figure 16: Comparison of image
fusion performance using metrics ‘NMI’ to understand the impact of attention
mechanism.