# On Performance of Fluid Antenna System using Maximum Ratio Combining

Xiazhi Lai, Tuo Wu, Junteng Yao, Cunhua Pan, _Senior Member, IEEE_ ,
Maged Elkashlan, _Senior Member, IEEE_ , and Kai-Kit Wong, _Fellow, IEEE_
_(Corresponding author: Tuo Wu and Cunhua Pan.)_ X. Lai is with the School of
Computer Science, Guangdong University of Education, Guangzhou, Guangdong,
China (E-mail: xzlai@outlook.com).T. Wu and M. Elkashlan are with the School
of Electronic Engineering and Computer Science at Queen Mary University of
London, London E1 4NS, U.K. (Email:{tuo.wu, maged.elkashlan}@qmul.ac.uk).J.
Yao is with the Faculty of Electrical Engineering and Computer Science, Ningbo
University, Ningbo 315211, China (E-mail: juntengyao512@163.com).C. Pan is
with the National Mobile Communications Research Laboratory, Southeast
University, Nanjing 210096, China. (e-mail: cpan@seu.edu.cn).K. K. Wong is
with the Department of Electronic and Electrical Engineering, University
College London, WC1E 6BT London, U.K., and also with the Yonsei Frontier
Laboratory and the School of Integrated Technology, Yonsei University, Seoul
03722, South Korea (e-mail: kat-kit.wong@ucl.ac.uk).

###### Abstract

This letter investigates a fluid antenna system (FAS) where multiple ports can
be activated for signal combining for enhanced receiver performance. Given
$\displaystyle M$ ports at the FAS, the best $\displaystyle K$ ports out of
the $\displaystyle M$ available ports are selected before maximum ratio
combining (MRC) is used to combine the received signals from the selected
ports. The aim of this letter is to study the achievable performance of FAS
when more than one ports can be activated. We do so by analyzing the outage
probability of this setup in Rayleigh fading channels through the utilization
of Gauss-Chebyshev integration, lower bound estimation, and high signal-to-
noise ratio (SNR) asymptotic approximations. Our analytical results
demonstrate that FAS can harness rich spatial diversity, which is confirmed by
computer simulations.

###### Index Terms:

Diversity, fluid antenna system (FAS), maximum ratio combining (MRC), outage
probability.

##  I Introduction

Fluid antenna system (FAS) capitalizes upon the inherent spatial diversity by
dynamically adjusting the antenna elements to optimal positions, referred to
as “ports”. This new paradigm stands in contrast to traditional communication
methodologies, in which the antenna elements remain in fixed positions, as
elucidated by Shojaeifard et al. in [1]. The realization of FAS may come in
the forms of liquid-metal-based antennas [2] or on-off pixel-based antennas
[3]. See [4] for more details.

Motivated by the great potential of FAS, recent research has delved into the
FAS channel model, deriving the probability density function (PDF) of the
received signal-to-noise ratio (SNR) as well as the corresponding outage
probability [5, 6, 7]. Remarkably, the outcomes of their investigation
unveiled the superiority of the FAS scheme over conventional fixed-position
antenna systems, particularly when a considerable multitude of ports is at
disposal. Machine learning techniques have also been shown to be effective in
port selection for FAS [8]. Most recently, Wong _et al._ has extended the use
of FAS for multiple access by taking advantage of the ups and downs of fading
channels in the spatial domain, and illustrated the possibility of alternative
multiple access schemes using FAS [9, 10, 11].

However, research in FAS is still in an early stage and the majority of the
results so far are limited to FAS with only one selected port exhibiting the
maximum SNR [5, 6, 7, 8, 9, 10]. The fact that a mobile terminal can actually
afford more than one radio frequency (RF) chains, means that it is
increasingly probable that FAS can come with multiple activated ports, with
better performance [12]. Since maximum ratio combining (MRC) is the optimal
mixing scheme without interference, it is therefore of great importance to
understand the achievable performance of FAS using MRC if more than one ports
can be selected for reception. This is the aim of this letter.

Specifically, our contributions are summarized as follows:

  * •

First, we consider a $\displaystyle K$-port FAS which corresponds to a FAS
with $\displaystyle K$ selected ports, operating in Rayleigh fading channels.
The mobile receiver selectively activates $\displaystyle K$ optimal ports from
the available $\displaystyle M$ ports. Then MRC is employed to combine the
$\displaystyle K$ branches of signals from the activated ports. We derive the
outage probability of the proposed $\displaystyle K$-port FAS using both
Laplace transform (LT) and Gauss-Chebyshev integration methods.

  * •

Additionally, we present the lower bound and asymptotic expressions for the
outage probability.

  * •

The simulation results substantiate the effectiveness of the proposed
analytical approach, thereby confirming and validating our insights and
discussions.

##  II System Model

Consider an end-to-end communication in Rayleigh fading channels, where the
source transmits the signal using a conventional fixed-position antenna with
transmit power $\displaystyle P_{S}$ but the receiver is equipped with a FAS
with $\displaystyle K$ fluid antenna elements.111In our idealized mathematical
model, a FAS with multiple single-activated-port fluid antennas is equivalent
to a FAS with multiple activated ports although their specific implementation
details will differ. Each antenna element is connected to one RF chain. Within
this particular FAS configuration, a linear space of $\displaystyle W\lambda$
encompasses a total of $\displaystyle M$ ports, where $\displaystyle\lambda$
represents the wavelength [5]. Among these $\displaystyle M$ ports, it is
assumed that each port is evenly distributed, and $\displaystyle K$ ports can
be activated for signal receiving out of the total $\displaystyle M$ ports.

Since each port is placed closely, the channel parameters of each port are
correlated. Building upon the channel model developed in [7] and [10], we
introduce a virtual reference port to model the channel correlation. This
virtual reference port is characterized by a channel parameter $\displaystyle
h_{0}\sim\mathcal{CN}(0,\alpha)$, following a complex Gaussian distribution
with zero mean and variance $\displaystyle\alpha$. Accordingly, the SNR of
$\displaystyle h_{0}$ can be written as

$\displaystyle\displaystyle\gamma_{0}=\frac{P_{S}|h_{0}|^{2}}{\sigma^{2}},$
(1)

where $\displaystyle\sigma^{2}$ denotes the noise power level. Considering
$\displaystyle h_{0}$ as a complex Gaussian random variable (RV), the PDF of
$\displaystyle\gamma_{0}$ can be expressed as

$\displaystyle\displaystyle
f_{\gamma_{0}}(x)=\frac{1}{\phi}e^{-\frac{x}{\phi}},$ (2)

where $\displaystyle\phi=P_{S}\alpha/\sigma^{2}$ represents the average
received SNR. Now, we proceed to establish the channel parameter linking the
source and the $\displaystyle m$-th port, denoted as $\displaystyle h_{m}$,
where $\displaystyle m\in\mathcal{M}=\\{1,2,\dots,M\\}$. The expression for
$\displaystyle h_{m}$ takes the form

$\displaystyle\displaystyle h_{m}=\mu h_{0}+(1-\mu)e_{m},$ (3)

where $\displaystyle e_{m}\sim\mathcal{CN}(0,\alpha)$ for $\displaystyle
m\in\mathcal{M}$ are independently and identically distributed (i.i.d.) RVs,
$\displaystyle\alpha$ is the average channel gain from the source to the
ports. Additionally, $\displaystyle\mu$ denotes the correlation factor, which
is given by [7]

$\displaystyle\displaystyle\mu=$
$\displaystyle\displaystyle\sqrt{2}\sqrt{{}_{1}F_{2}\Big{(}\frac{1}{2};1;\frac{3}{2};-\pi^{2}W^{2}\Big{)}-\frac{J_{1}(2\pi
W)}{2\pi W}},$ (4)

where $\displaystyle{}_{a}F_{b}$ denotes the generalized hypergeometric
function and $\displaystyle J_{1}(\cdot)$ is the first-order Bessel function
of the first kind.

Conditioned on a fixed channel parameter $\displaystyle h_{0}$, and in
accordance with $\displaystyle\gamma_{0}$, the corresponding SNR of
$\displaystyle h_{m}$, expressed as
$\displaystyle\gamma_{m}=\frac{P_{S}|h_{m}|^{2}}{\sigma^{2}}$, follows a non-
central chi-square distribution. The conditional PDF can be expressed as

$\displaystyle\displaystyle f_{\gamma_{m}|\gamma_{0}=x_{0}}(x)=$
$\displaystyle\displaystyle\omega e^{-\omega(x+\mu
x_{0})}I_{0}\big{(}2\omega\sqrt{\mu x_{0}x}\big{)},$ (5)

where $\displaystyle\omega=\big{(}\phi(1-\mu)\big{)}^{-1}$. Besides,
$\displaystyle I_{0}(u)$ is the modified Bessel function of the first kind
with order $\displaystyle 0$, which can be expressed in series representation
as [13]

$\displaystyle\displaystyle
I_{0}(z)=\sum_{k=0}^{\infty}\frac{z^{2k}}{2^{2k}k!\Gamma(k+1)}.$ (6)

Combining (5) with (6), we further derive $\displaystyle
f_{\gamma_{m}|\gamma_{0}=x_{0}}(x)$ as

$\displaystyle\displaystyle f_{\gamma_{m}|\gamma_{0}=x_{0}}(x)=$
$\displaystyle\displaystyle\sum_{k=0}^{\infty}c_{k}x_{0}^{k}e^{-\omega\mu
x_{0}}x^{k}e^{-\omega x},$ (7)

where

$\displaystyle\displaystyle c_{k}=\frac{\omega^{2k+1}\mu^{k}}{(k!)^{2}}.$ (8)

In order to receive the signal transmitted from the source, the receiver
selects the $\displaystyle K$ ports with the $\displaystyle K$ highest
received SNR from the available total of $\displaystyle M$ ports for
activation. The set of selected ports is denoted by

$\displaystyle\displaystyle\mathbb{K}=\arg\emph{\mbox{K}}\max_{m\in\mathcal{M}}\gamma_{m},$
(9)

where $\displaystyle\emph{\mbox{K}}\max_{m\in\mathcal{M}}\gamma_{m}$ denotes
to select the $\displaystyle K$ maximal $\displaystyle\gamma_{m}$ out of set
$\displaystyle\mathcal{M}$. In addition, to process the received signals from
different antenna elements, the MRC technique is utilized to combine the
$\displaystyle K$ branches of signals.

Moreover, the channel state information (CSI) is assumed to be not available
at the source; hence the transmission data rate is fixed to $\displaystyle R$.
Therefore, the outage of communication occurs when the FAS cannot sustain the
data rate $\displaystyle R$, i.e.,

$\displaystyle\displaystyle\mathcal{E}=\left\\{\log_{2}\left(1+\sum_{m\in\mathbb{K}}\gamma_{m}\right)\leq
R\right\\}.$ (10)

Thus, the system’s outage probability is written as

$\displaystyle\displaystyle P_{\rm{out}}=\Pr\left(\mathcal{E}\right).$ (11)

##  III Performance Analysis

Here, we derive the exact outage probability of the proposed FAS-enabled
communications. Subsequently, the lower bound and asymptotic expressions of
the outage provability of system are derived. These derivations offer valuable
insights for the proposed FAS-enabled communications system.

###  III-A Exact Outage Probability

Consider the port with the $\displaystyle(K+1)$-th maximal channel gain,
denoted as $\displaystyle v$. Given $\displaystyle\gamma_{0}=x_{0}$, the
outage probability is expressed as

$\displaystyle\displaystyle\Lambda(z)$ $\displaystyle\displaystyle\hskip
1.42262pt=\Pr\left(\sum_{m\in\mathbb{K}}\gamma_{m}\leq
z|\gamma_{0}=x_{0}\right)$
$\displaystyle\displaystyle\overset{(a)}{=}\binom{M}{K}(T+1)\int_{0}^{\infty}\Phi(z)\Psi(v)f_{v|\gamma_{0}=x_{0}}(v)dv,$
(12)

where $\displaystyle z=2^{R}-1$ denotes the SNR threshold of outage,
$\displaystyle T=M-K-1$, and

$\displaystyle\displaystyle\Psi(v,x_{0})=\Pr\left(\gamma_{m}\leq
v,m\in\mathcal{T}|\gamma_{0}=x_{0}\right),$ (13)

is the probability that $\displaystyle T+1$ ports are idle with maximal
channel gain $\displaystyle v$, and
$\displaystyle\mathcal{T}=\\{1,2,\dots,T\\}$. Also,

$\displaystyle\displaystyle\Phi(z,v,x_{0})=\Pr\left(\sum_{m\in\mathcal{K}}\gamma_{m}\leq
z,\gamma_{m}>v|\gamma_{0}=x_{0}\right),$ (14)

is the probability that $\displaystyle K$ ports are selected and outage occurs
and $\displaystyle\mathcal{K}=\\{1,2,\dots,K\\}$. Step ($\displaystyle a$)
holds since $\displaystyle\gamma_{m}$ for $\displaystyle m\in\mathcal{M}$ are
i.i.d. RVs, and $\displaystyle\Psi(v,x_{0})\Phi(z,v,x_{0})$ represents the
outage probability related to one of the port selection results.

In the following, we derive the expressions of $\displaystyle\Psi(v,x_{0})$
and $\displaystyle\Phi(z,v,x_{0})$. Then we obtain the outage probability by
taking the expectation of $\displaystyle\Lambda(z)$ with respect to
$\displaystyle\gamma_{0}$.

First, it is important to note that $\displaystyle\forall m,l\in\cal{T}$,
$\displaystyle\gamma_{m}$ and $\displaystyle\gamma_{l}$ are independent with
each other given $\displaystyle\gamma_{0}=x_{0}$. Furthermore, in accordance
with (5), the joint PDF of $\displaystyle\gamma_{m}$ for $\displaystyle
m\in\cal{T}$ can be expressed as

$f_{\gamma_{m},m\in\mathcal{T}|\gamma_{0}=x_{0}}(x_{1},\dots,x_{T})\\\
=\prod_{m=1}^{T}\omega e^{-\omega(x_{m}+\mu
x_{0})}I_{0}\big{(}2\omega\sqrt{\mu x_{0}x_{m}}\big{)}.$ (15)

Then, by utilizing (13) and (15), we evaluate $\displaystyle\Psi(v,x_{0})$ as

$\displaystyle\displaystyle\Psi(v,x_{0})$
$\displaystyle\displaystyle=\int^{v}_{0}\cdots\int^{v}_{0}f_{\gamma_{m},m\in\mathcal{T}|\gamma_{0}=x_{0}}(x_{1},\dots,x_{T})dx_{1}\cdots
dx_{T}$ $\displaystyle\displaystyle=\Big{(}1-Q_{1}\big{(}\sqrt{2\omega\mu
x_{0}},\sqrt{2\omega v}\big{)}\Big{)}^{T},$ (16)

where $\displaystyle Q_{1}(\cdot,\cdot)$ is the first order
Marcum-$\displaystyle Q$ function [6].

Next, we proceed to derive the analytical expression of
$\displaystyle\Phi(z,v,x_{0})$ by utilizing the following theorem.

###### Theorem 1

The LT expressions of the following functions

$\displaystyle\displaystyle g(x)$
$\displaystyle\displaystyle=x^{a}e^{-bx}u(x-v),$ (17)
$\displaystyle\displaystyle p(x)$
$\displaystyle\displaystyle=(x-a)^{K-1}e^{-bx}u(x-a),$ (18)

are, respectively,

$\displaystyle\displaystyle L[g(x);s]$
$\displaystyle\displaystyle=e^{-(s+b)v}\sum_{l=0}^{a}\frac{a!v^{l}}{l!(s+b)^{a+1-l}},$
(19) $\displaystyle\displaystyle L[p(x);s]$
$\displaystyle\displaystyle=\frac{(K-1)!e^{-a(s+b)}}{(s+b)^{K}},$ (20)

where $\displaystyle\mathrm{Re}(s)\geq-b$, $\displaystyle\rm{Re}(x)$ denotes
the real part of $\displaystyle x$, and $\displaystyle u(\cdot)$ is the step
function.

###### Proof 1

See Appendix A.

From Theorem 1 and (7), the LT of the PDF of $\displaystyle\gamma_{m}$ with
$\displaystyle\gamma_{m}>v$ is given by

$L\big{[}f_{\gamma_{m}|\gamma_{0}=x_{0}}(x_{m});s\big{]}\\\
=e^{-(s+\omega)v-\omega\mu
x_{0}}\sum_{m=0}^{\infty}\sum_{l=0}^{m}\frac{d_{m}x_{0}^{m}v^{l}}{l!(s+\omega)^{m+1-l}},$
(21)

where $\displaystyle\rm{Re}(s)\geq-\omega$ and $\displaystyle d_{m}=c_{m}m!$.

Then, by using the faltung theorem in [13], the LT of the PDF of RV
$\displaystyle\bar{\gamma}=\sum_{m=1}^{K}\gamma_{m}$ conditioned on
$\displaystyle\gamma_{m}>v$ can be derived as

$\displaystyle\displaystyle
L\big{[}f_{\bar{\gamma}|\gamma_{0}=x_{0}}(x);s\big{]}$
$\displaystyle\displaystyle=\Big{(}L\big{[}f_{\gamma_{m}|\gamma_{0}=x_{0}}(x_{m});s\big{]}\Big{)}^{K}$
$\displaystyle\displaystyle=e^{-Kv(s+\omega)-K\omega\mu
x_{0}}\sum_{r_{m}=0\atop
m\in\mathcal{K}}^{\infty}\rho_{m}x_{0}^{\eta_{m}}\sum_{l_{m}=0\atop
m\in\mathcal{K}}^{r_{m}}\frac{v^{\epsilon_{m}}q_{m}}{(s+\omega)^{\chi_{m}}},$
(22)

where

$\left\\{\begin{aligned} \rho_{m}&=\prod_{m=1}^{K}d_{m},\\\
\eta_{m}&=\sum_{m=1}^{K}r_{m},\\\ \epsilon_{m}&=\sum_{m=1}^{K}l_{m},\\\
q_{m}=&\prod_{m=1}^{K}\frac{1}{l_{m}!},\\\
\chi_{m}&=K+\eta_{m}-\epsilon_{m}.\end{aligned}\right.$ (23)

Utilizing Theorem 1, we can obtain the PDF of $\displaystyle\bar{\gamma}$
conditioned on $\displaystyle\gamma_{0}=x_{0}$ as

$f_{\bar{\gamma}|\gamma_{0}=x_{0}}(x)=e^{-\omega(x+K\mu
x_{0})}\sum_{r_{m}=0\atop m\in\mathcal{K}}^{\infty}\rho_{m}x_{0}^{\eta_{m}}\\\
\times\sum_{l_{m}=0\atop
m\in\mathcal{K}}^{r_{m}}v^{\epsilon_{m}}q_{m}\frac{\big{(}x-Kv\big{)}^{\chi_{m}-1}}{(\chi_{m}-1)!},$
(24)

with $\displaystyle x\geq Kv$. Based on (24), the computation of
$\displaystyle\Phi(z,v,x_{0})$ can be performed by

$\displaystyle\displaystyle\Phi(z,v,x_{0})$
$\displaystyle\displaystyle=\int_{Kv}^{z}f_{\bar{\gamma}|\gamma_{0}=x_{0}}(x)dx$
$\displaystyle\displaystyle=e^{-\omega(Kv+K\mu x_{0})}\sum_{r_{m}=0\atop
m\in\mathcal{K}}^{\infty}\rho_{m}x_{0}^{\eta_{m}}$
$\displaystyle\displaystyle\quad\times\sum_{l_{m}=0\atop
m\in\mathcal{K}}^{r_{m}}v^{\epsilon_{m}}q_{m}\frac{\gamma\big{(}\chi_{m},\omega(z-Kv)\big{)}}{(\chi_{m}-1)!\omega^{\chi_{m}}},$
(25)

in which $\displaystyle z\geq Kv$ is a necessary condition; otherwise,
$\displaystyle\Phi(z,v,x_{0})=0$. In addition, $\displaystyle\gamma(\alpha,x)$
is the lower incomplete Gamma function, which can be expressed in integral and
serial representations respectively, as

$\displaystyle\displaystyle\gamma(\kappa,x)=\int_{0}^{x}e^{-t}t^{\kappa-1}dt=(\kappa-1)!\left(1-e^{-x}\sum_{m=0}^{\kappa-1}\frac{x^{m}}{m!}\right).$
(26)

Calculating $\displaystyle\Lambda(z)$ in (III-A) with (III-A) and (III-A), and
then taking the expectation of $\displaystyle\Lambda(z)$ with respect to
$\displaystyle\gamma_{0}$, the outage probability of the system can be
computed as

$P_{\mathrm{out}}=\int_{0}^{\infty}\int_{0}^{\frac{z}{K}}\binom{M}{K}(T+1)\Phi(z,v,x_{0})\Psi(v,x_{0})\\\
\times f_{\gamma_{m}|\gamma_{0}=x_{0}}(v)f_{\gamma_{0}}(x_{0})dvdx_{0}.$ (27)

###### Remark 1

From (III-A), it becomes evident that $\displaystyle\Psi(v,x_{0})$ becomes
tiny with a large number of $\displaystyle T$, owing to the fact that
$\displaystyle Q_{1}(\cdot,\cdot)$ is bounded by 1 [6]. This observation
implies that $\displaystyle P_{\mathrm{out}}$ in (27), i.e., the outage
probability of the system approaches zero when the total number of ports
$\displaystyle M\rightarrow\infty$.222Note that the conclusion may vary
depending on how spatial correlation over the ports is modelled. That said,
the analysis presented in this letter gives the first-look performance of FAS
using MRC.

It is noticeable that the integral in (27) presents computational challenges.
To address this, we initially replace the upper limit of the integral in (27)
with a sufficiently large value denoted as $\displaystyle H$. This
approximation is valid because the integrand in (27) tends to approach zero as
$\displaystyle x_{0}$ increases. Subsequently, we resort to the Gauss-
Chebyshev integral to derive a precise approximation of $\displaystyle
P_{\mathrm{out}}$ in serial representation:

$P_{\mathrm{out}}\approx\binom{M}{K}\frac{\pi^{2}Hz(T+1)}{4U_{p}U_{l}}\sum_{p=1}^{U_{p}}\sum_{l=1}^{U_{l}}\Phi(z,y_{l},y_{p})\Psi(y_{l},y_{p})\\\
\times\sqrt{1-t_{p}^{2}}\sqrt{1-t_{l}^{2}}f_{\gamma_{m}|\gamma_{0}=y_{p}}(y_{l})f_{\gamma_{0}}(y_{p}),$
(28)

where $\displaystyle U_{p}$ and $\displaystyle U_{l}$ are complexity-accuracy
tradeoff parameters, and

$\left\\{\begin{aligned} t_{p}&=\cos\left(\frac{(2p-1)\pi}{2U_{p}}\right),\\\
y_{p}&=\frac{H(t_{p}+1)}{2},\\\
t_{l}&=\cos\left(\frac{(2l-1)\pi}{2U_{l}}\right),\\\
y_{l}&=\frac{z(t_{l}+1)}{2K}.\end{aligned}\right.$ (29)

According to [14], it is established that the approximation provided in (28)
is tight with large numbers of $\displaystyle U_{p}$ and $\displaystyle
U_{l}$.

###  III-B Lower Bound and Asymptotic Analysis

For the sake of facilitating computation and analysis of $\displaystyle
P_{\rm{out}}$, we derive a lower bound for $\displaystyle P_{\rm{out}}$ in
this subsection. Notably, this lower bound closely approximates the exact
outage probability, particularly in the high SNR region. Moreover, we analyze
the asymptotic behavior of $\displaystyle P_{\rm{out}}$ and discuss the
performance bottleneck of the system.

First, from (7), we can readily know that $\displaystyle
f_{\gamma_{m}|\gamma_{0}=x_{0}}(x)$ is lower-bounded by

$\displaystyle\displaystyle\bar{f}_{\gamma_{m}|\gamma_{0}=x_{0}}(x)=\omega
e^{-\omega\mu x_{0}}e^{-\omega x}.$ (30)

Based on (30), we can accordingly obtain the lower bound of
$\displaystyle\Psi(v,x_{0})$ and $\displaystyle\Phi(z,v,x_{0})$, respectively,
as

$\displaystyle\displaystyle\bar{\Psi}(v,x_{0})$
$\displaystyle\displaystyle=e^{-\omega T\mu
x_{0}}\sum_{t=0}^{T}\binom{T}{t}(-1)^{t}e^{-\omega tv},$ (31)
$\displaystyle\displaystyle\bar{\Phi}(z,v,x_{0})$
$\displaystyle\displaystyle=e^{-\omega(Kv+K\mu x_{0})}-e^{-\omega(z+K\mu
x_{0})}$
$\displaystyle\displaystyle\times\sum_{k=0}^{K-1}\frac{\omega^{k}}{k!}\sum_{m=0}^{k}\binom{k}{m}z^{k-m}(-Kv)^{k}.$
(32)

Applying (30)–(III-B) into (27), we can obtain

$\displaystyle\displaystyle P_{\rm{out}}$
$\displaystyle\displaystyle\geq\bar{P}_{\rm{out}}$
$\displaystyle\displaystyle=\binom{M}{K}\frac{T+1}{M\mu\omega\phi+1}$
$\displaystyle\displaystyle\times\left(\sum_{t=0}^{T}\binom{T}{t}\beta_{t}-\sum_{t=0}^{T}\sum_{k=0}^{K-1}\sum_{m=0}^{k}\binom{T}{t}\binom{k}{m}\kappa_{t,k,m}\right),$
(33)

where

$\displaystyle\displaystyle\beta_{t}$
$\displaystyle\displaystyle=\frac{(-1)^{t}}{(t+K+1)}\big{(}1-e^{-z\omega(t+K+1)}\big{)},$
(34) $\displaystyle\displaystyle\kappa_{t,k,m}$
$\displaystyle\displaystyle=\frac{(-1)^{t+m}K^{m}(z\omega)^{k-m}\gamma\left(m+1,\frac{z\omega(t+1)}{K}\right)}{k!(t+1)^{m+1}}.$
(35)

From (7), it becomes apparent that the exact value of $\displaystyle
P_{\rm{out}}$ approaches the lower bound $\displaystyle\bar{P}_{\rm{out}}$ as
the average received SNR becomes large, i.e., the values of
$\displaystyle\alpha$ or $\displaystyle P_{S}$ are large. Moreover, when the
value of $\displaystyle M$ and $\displaystyle K$ are large, the calculations
of outage probabilities in (27) and (28) become intricate. In contrast, the
evaluation of $\displaystyle\bar{P}_{\rm{out}}$ using the expression in
(III-B) remains computationally efficient, aiding in the analysis of the
performance of the proposed system.

Furthermore, by applying the expansion $\displaystyle e^{-x}=1-x$ for tiny
value of $\displaystyle|x|$, we can obtain the asymptotic expression of outage
probability in the high SNR region as

$\displaystyle\displaystyle P_{\rm{out}}\simeq\psi(z\omega)^{M},$ (36)

where

$\psi=\\\
\binom{M}{K}\frac{(T+1)(1-\mu)}{K!(M\mu+1-\mu)K^{T+1}}\sum_{k=0}^{K}\binom{K}{k}\frac{1}{k+T+1}.$
(37)

###### Remark 2

The asymptotic outage probability in (36) indicates that the diversity order
of the FAS-aided communication system is $\displaystyle M$. This means that
the proposed system can fully exploit the diversity offered by total available
$\displaystyle M$ ports, regardless of the number of activated ports
$\displaystyle K$. Therefore, enhancing the system’s performance by increasing
the number of $\displaystyle K$ ports is feasible; yet the improvement is less
significant than the improvement of increasing $\displaystyle M$.

##  IV Numerical Results

In this section, we present several numerical results for the FAS-aided
communications. Following a similar approach to the work in [9], we assume the
value of $\displaystyle W=5$, which is a common choice for 5G networks in the
context of handset devices. Moreover, we set the data rate $\displaystyle R$
to $\displaystyle 5$ bit/s/Hz, leading to an outage SNR threshold
$\displaystyle z$ set at 31. Unless specified otherwise, we refer to the
outcomes of our simulations as “Simul”. Also, we denote the results obtained
from (28), (III-B), and (36) as “Ana.”, “LB”, and “Asy.”, respectively.

Figure 1: Outage probability versus average SNR $\displaystyle\phi$.

Fig. 1 illustrates the variations in outage probability with the average SNR
($\displaystyle\phi$), considering different values of $\displaystyle M$ and
$\displaystyle K$. As observed from Fig. 1, it is evident that the analytical
outage probability derived from equation (28) closely aligns with the
simulation results. Additionally, the lower bound provided by equation (III-B)
accurately approximates the simulation outcomes, particularly in the high SNR
region, which corroborates with the asymptotic result in equation (36).

Furthermore, Fig. 1 indicates that the outage probability of the system is
predominantly influenced by the total port number $\displaystyle M$, affirming
the analysis in equation (36) that the diversity stemming from all available
ports can be maximally exploited. It is worth noting that the gain achieved by
increasing $\displaystyle K$ from $\displaystyle 2$ to $\displaystyle 4$ in
the high SNR region is approximately $\displaystyle 3.8$, consistent with the
findings presented in equation (36). However, the enhancement resulting from
increasing the number of activated ports $\displaystyle K$ is comparatively
less pronounced than the gains derived from increasing $\displaystyle M$.

Figure 2: Outage probability versus number of antenna elements $\displaystyle
K$.

Fig. 2 provides a visualization of the relationship between the number of
activated ports ($\displaystyle K$) and the resulting outage probability in
the context of the $\displaystyle K$-port FAS-aided communications system. The
experiment is conducted with $\displaystyle\phi$ set at $\displaystyle 10$ dB,
and two distinct values for the total port count ($\displaystyle M$), namely
$\displaystyle 10$ and $\displaystyle 20$. Meanwhile, the number of activated
ports $\displaystyle K$ is allowed to vary within the interval of
$\displaystyle 1$ to $\displaystyle 8$.

Upon examining the results depicted in Fig. 2, it becomes evident that
increasing the count of activated ports ($\displaystyle K$) contributes
significantly to enhancing the overall system’s outage performance. However,
the most striking insight emerges from the clear trend indicating that the
advantages stemming from augmenting the total port count ($\displaystyle M$)
are even more pronounced. This noteworthy pattern is in concordance with the
analytical findings presented in the preceding sections.

##  V Conclusion

In this letter, we proposed to analyze the FAS-aided communications system
with multiple activated ports, where the MRC technique was utilized to combine
the signal from different activated ports. The outage probability of the
proposed system has been derived in Rayleigh fading channels, in forms of
exact expression, lower bound, and asymptotic expression. Analysis showed that
the diversity order of the system equals the number of total available ports.
Simulation results corroborated the effectiveness of the provided analysis.

##  Appendix A Proof of Theorem 1

According to the definition of LT, we can compute the LT expression of
$\displaystyle g(x)$ in (17) as

$\displaystyle\displaystyle L[g(x);s]$
$\displaystyle\displaystyle=\int_{0}^{\infty}g(x)e^{-sx}dx=\int_{v}^{\infty}x^{a}e^{-(s+b)x}dx$
$\displaystyle\displaystyle=e^{-(s+b)v}\sum_{l=0}^{a}\frac{a!v^{l}}{l!(s+b)^{a+1-l}},$
(38)

where $\displaystyle\mathrm{Re}(s)\geq-b$, and the last step can be derived by
using the partial integral technique.

Similarly, we can compute the LT expression of $\displaystyle p(x)$ as

$\displaystyle\displaystyle L[p(x);s]$ $\displaystyle\displaystyle\hskip
2.84526pt=\int_{0}^{\infty}p(x)e^{-sx}dx$ $\displaystyle\displaystyle\hskip
2.84526pt=\int_{a}^{\infty}(x-a)^{K-1}e^{-(b+s)x}dx$
$\displaystyle\displaystyle\overset{(e_{1})}{=}\frac{e^{-a(s+b)}}{(s+b)^{K}}\int_{0}^{\infty}t^{K-1}e^{-t}dt$
$\displaystyle\displaystyle\overset{(e_{2})}{=}\frac{(K-1)!e^{-a(s+b)}}{(s+b)^{K}},$
(39)

where $\displaystyle\mathrm{Re}(s)\geq-b$, the step ($\displaystyle e_{1}$)
can be obtained by setting $\displaystyle t=(x-a)(s+b)$, and step
($\displaystyle e_{2}$) uses the partial integral technique.

## References

  * [1] A. Shojaeifard et al., “MIMO evolution beyond 5G through reconfigurable intelligent surfaces and fluid antenna systems,” _Proc. IEEE_ , vol. 110, no. 9, pp. 1244–1265, Sept. 2022.
  * [2] Y. Huang, L. Xing, C. Song, S. Wang, and F. Elhouni, “Liquid antennas: Past, present and future,” _IEEE Open J. Antennas Propag._ , vol. 2, pp. 473–487, 2021.
  * [3] D. Rodrigo, B. A. Cetiner, and L. Jofre, “Frequency, radiation pattern and polarization reconfigurable antenna using a parasitic pixel layer,” _IEEE Trans. Antennas Propag._ , vol. 62, no. 6, pp. 3422–3427, Jun. 2014.
  * [4] K. K. Wong, K. F. Tong, Y. Shen, Y. Chen, and Y. Zhang, “Bruce Lee-inspired fluid antenna system: Six research topics and the potentials for 6G,” Frontiers Commun. Netw., section Wireless Commun., 3:853416, Mar. 2022.
  * [5] K.-K. Wong, A. Shojaeifard, K.-F. Tong and Y. Zhang, “Performance limits of fluid antenna systems,” _IEEE Commun. Letters_ , vol. 24, no. 11, pp. 2469–2472, Nov. 2020.
  * [6] K.-K. Wong, A. Shojaeifard, K.-F. Tong, and Y. Zhang, “Fluid antenna systems,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 3, pp. 1950–1962, March 2021.
  * [7] K.-K. Wong, K. F. Tong, Y. Chen, and Y. Zhang, “Closed-form expressions for spatial correlation parameters for performance analysis of fluid antenna systems,” _IET Electron. Lett._ , vol. 58, no. 11, pp. 454–457, May 2022.
  * [8] Z. Chai, K.-K. Wong, K.-F. Tong, Y. Chen, and Y. Zhang, “Port selection for fluid antenna systems,” _IEEE Commun. Lett._ , vol. 26, no. 5, pp. 1180–1184, May 2022.
  * [9] K.-K. Wong and K.-F. Tong, “Fluid antenna multiple access,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 7, pp. 480C4815, Jul. 2022.
  * [10] K.-K. Wong, D. Morales-Jimenez, K.-F. Tong, and C.-B. Chae, “Slow fluid antenna multiple access,” _IEEE Trans. Commun._ , vol. 71, no. 5, pp. 2831–2846, May 2023.
  * [11] N. Waqar, K.-K. Wong, K.-F. Tong, A. Sharples, and Y. Zhang, “Deep learning enabled slow fluid antenna multiple access,” _IEEE Commun. Lett._ , vol. 27, no. 3, pp. 861–865, March 2023.
  * [12] K.-K. Wong, K.-F. Tong, and C.-B. Chae, “Fluid antenna system-part II: Research opportunities,” _IEEE Commun. Lett._ , vol. 27, no. 8, pp. 1924–1928, Aug. 2023
  * [13] I. S. Gradshteyn and I. M. Ryzhik, _Table of Integrals, Series, and Products, 7th ed_. San Diego, CA: Academic, 2007
  * [14] E. Suli and D. F. Mayers, _An Introduction to Numerical Ananlysis_. Cambridge, U.K.: Cambridge Univ. Press, 2003.


∎

11institutetext: C. Cheng, T. Xu, X. J. Wu*, and H. Li 22institutetext: School
of Artificial Intelligence and Computer Science
Jiangnan University, Wuxi, 214122, China.
22email<EMAIL_ADDRESS>33institutetext: X. Li 44institutetext:
College of Computer Science and Technology
Zhejiang University, Hangzhou, 310027, China.
44email<EMAIL_ADDRESS>55institutetext: J. Kitter 66institutetext: Centre
for Vision, Speech and Signal Processing
University of Surrey, Guildford, GU2 7XH, UK.
66email<EMAIL_ADDRESS>
# FusionBooster: A Unified Image Fusion Boosting Paradigm

Chunyang Cheng  Tianyang Xu  Xiao-Jun Wu  Hui Li  Xi Li  Josef Kittler

(Received: July, 2023 / Accepted: date)

###### Abstract

In recent years, numerous ideas have emerged for designing a mutually
reinforcing mechanism or extra stages for the image fusion task, ignoring the
inevitable gaps between different vision tasks and the computational burden.
We argue that there is a scope to improve the fusion performance with the help
of the FusionBooster, a model specifically designed for the fusion task. In
particular, our booster is based on the divide-and-conquer strategy controlled
by an information probe. The booster is composed of three building blocks: the
probe units, the booster layer, and the assembling module. Given the result
produced by a backbone method, the probe units assess the fused image and
divide the results according to their information content. This is
instrumental in identifying missing information, as a step to its recovery.
The recovery of the degraded components along with the fusion guidance are the
role of the booster layer. Lastly, the assembling module is responsible for
piecing these advanced components together to deliver the output. We use
concise reconstruction loss functions in conjunction with lightweight
autoencoder models to formulate the learning task, with marginal computational
complexity increase. The experimental results obtained in various fusion
tasks, as well as downstream detection tasks, consistently demonstrate that
the proposed FusionBooster significantly improves the performance. Our code
will be publicly available at https://github.com/AWCXV/FusionBooster.

###### Keywords:

Image fusion Unified Lightweight Booster.

##  1 Introduction

Image fusion is a technique aiming to combine complementary information from
diverse modalities, or images with different shooting settings, into a single
image. The fused image, which becomes more informative, is expected to have
enhanced visual quality, as well as boost the performance of downstream vision
tasks. This technique has been widely applied to different areas, including
video surveillance, object tracking, remote sensing imaging, and medical
diagnosis (Xu et al., 2022a, 2019; Zhang, 2021; Tang et al., 2023b).

Broadly speaking, the current image fusion tasks fall into two main
categories, $i.e.$, multi-modal image fusion and digital photography fusion.
For instance, the infrared and visible image fusion (IVIF) task, which belongs
to the former category, arises in many practical applications. It aims to
combine the rich scene texture from the visible image, with the robust thermal
and structural information tapped from the infrared modality. Since the
infrared modality is insensitive to variations in the environmental condition,
combining these complementary sources of information helps to enhance the
visualization of challenging scenes, e.g., in the foggy or low-light
environments (Sun et al., 2022). On the other hand, multi-exposure image
fusion (MEIF) and multi-focus image fusion (MFIF) belong to the latter
category (digital photography). Specifically, the MEIF task is to combine the
input overexposed and underexposed images in order to generate fusion results
with an appropriate exposure setting (Xu et al., 2020b). The goal of the MFIF
task is to produce a fully focused image by combining the near-focused and
far-focused images at the input to counteract the depth-of-field limitation in
imaging (Zhang et al., 2021).

Figure 1:  Comparison of the proposed FusionBooster and other advanced methods
that contain additional enhancement models. The current algorithms are
suffering from the issues of expensive computational cost, task gap and the
lack of generalization ability. (Backbone method: DDcGAN (Ma et al., 2020a))

In the primary exploration stage, various signal processing techniques had
been applied to accomplish the fusion process in the conventional paradigm
exemplified by (Ma et al., 2016; Liu et al., 2016; Yang et al., 2018; Li et
al., 2020c, a; Chen et al., 2021). However, the limitations of the classical
feature extraction and fusion techniques motivated the emergence of deep
learning-based fusion methods (Li and Wu, 2018; Xu et al., 2020a; Zhang and
Ma, 2021; Tang et al., 2022b; Cheng et al., 2023). Currently, the trend has
shifted towards the focus on the interplay between fusion and other vision
tasks (Huang et al., 2022; Tang et al., 2023a, 2022a; Xu et al., 2022b). A few
studies also argue for adopting an extra training stage in the fusion task (Li
et al., 2021; Zhao et al., 2023b). However, as shown in Fig. 1, the
performance of current mainstream fusion methods is highly impeded by three
factors: the expensive computational overhead, the task gap, and the
inadequate generalization ability.

Specifically, the additional computational cost arises mainly from the other
vision models or extra training stages incorporated in their methods. Such
expensive overhead can hinder the practical adaptation of the fusion algorithm
to new scenes, when limited computational resources are available.
Furthermore, the introduction of other vision tasks also brings up the task
gap issue in the current image fusion paradigm. Typically, these methods
disregard the potential discrepancy between the nature of information
processing at low-level fusion, and high-level vision problems. Consequently,
the feedback from certain vision tasks may be completely inappropriate for the
task of refining the fusion model. That is, combining fusion and other vision
tasks with different objectives may result in suboptimal fused images. For
example, as illustrated in (a) of Fig. 1, the introduction of the low-light
enhancement model effectively improves the visible component of the fused
images, but it is not very effective at maintaining thermal radiation
information (the fusion task). Similarly, the result (b) also indicates that
the compatibility of the fusion and object detection tasks is not quite
satisfactory, as the visualization effect is not promising and the detection
precision is not significantly improved with the help of the detection model.
Finally, note that the current enhancement-based image fusion methods can only
work in a specific fusion task. The digital photography fusion tasks, i.e.,
the MFIF and the MEIF tasks, can not benefit from these paradigms, which
demonstrates their deficiency in the generalization ability.

Figure 2:  A comparison of the proposed divide and conquer boosting paradigm
(b) and existing methods (a) relying on the booster (other vision models). The
disentangled components allow us to better improve the fusion results in a
fine-grained manner, which also provides us with the flexibility to handle
more tasks, depending on the content.

In this work, we propose the FusionBooster model to address the above-
mentioned problems. Firstly, our network only consists of several
convolutional layers to formulate the encoder and decoder parts, forming a
compact model. This design can effectively alleviate the expensive
computational cost issue of existing enhancement-based methods.

Secondly, as our booster design reflects the characteristics of the image
fusion task, we do not require additional vision models to intervene in the
training process, thus avoiding the task gap issue. As shown in Fig. 2 (b),
given the initial fusion result, we specifically design an information probe
to reconstruct source images from it. If the assessment of the information
conveyed by the source image is of low quality, or the noise is introduced in
the first stage of processing, a satisfied quality reconstruction of the
source input from the fused image is generally impossible and the constituent
components tend to degrade. Interestingly, the degree of degradation is
correlated with the quality of the initial fusion results (Section 4.5.2).
Motivated by this observation, we incorporate, into the fusion system, a novel
mechanism (booster), which guides the process of reassembling these components
to produce the fused image. The mechanism enables the delivery of fusion
results which are more robust and of better quality. Compared with the
existing methods (Fig. 2 (a)), our FusionBooster succeeds in the
disentanglement of the fusion task and improves the fusion performance in a
fine-grained manner.

Thirdly, note that, depending on the characteristics of the fusion task, the
output of the information probes is different for, e.g., the overexposed and
underexposed components of the MEIF task. This content-specific focus allows
us to apply general operations on these detached components to benefit a
series of fusion tasks, which alleviates the lack of generalization in the
existing enhancement-based methods. More specifically, as our probe can be
regarded as a tool to gauge the information conveyed from the source images
(e.g., from the infrared and visible images) into the fusion results, we
design the corresponding booster layers to increase the information contained
in these separate components. In addition to this universal operation, we note
that, in some studies, the experimental analysis has shown that the salient
texture details can improve the performance of downstream vision tasks, as
well as produce visually pleasing fused images (Liu et al., 2022a; Cheng et
al., 2023). We argue that such enhancements can also consistently benefit
different fusion tasks. Thus, we take these findings into account in the
design of the booster layers. As a result, the upgraded methods can produce
fused images that are more robust and simultaneously preserve the significant
information from the source input to improve the performance of downstream
tasks.

The contributions of this work can be summarized as follows:

  * •

We devise an image fusion booster by analysing the quality of the initial
fusion results by means of a dedicated Information Probe.

  * •

In a new divide-and-conquer image fusion paradigm, the results of the analysis
performed by the Information Probe guide the refinement of the fused image
with the help of a nested autoencoder network.

  * •

The proposed FusionBooster is a general enhancer, which can be applied to
various image fusion methods, $e.g.$, traditional or learning-based
algorithms, irrespective of the type of fusion task.

  * •

The experimental results demonstrate that the proposed FusionBooster, in
general, significantly enhances the performance of the state-of-the-art (SOTA)
fusion methods and downstream detection tasks, with only a slight increase in
the computational overhead.

##  2 Related Work

###  2.1 Learning-based Image Fusion Methods

In recent years, various learning-based image fusion methods have been
proposed. These methods can be roughly divided into three categories, $i.e.$,
algorithms based on the generative adversarial networks (GAN), the
autoencoders (AE), and the regular convolutional neural networks (CNN).
Specifically, the GAN-based methods rely on the adversarial game established
between the generator and the discriminator to produce the fusion results (Fu
et al., 2021; Ma et al., 2020b). A representative work is the DDcGAN proposed
by Ma et al. (Ma et al., 2020a), which uses two discriminators to enable the
fused images to preserve the useful information from the infrared and visible
images.

Figure 3:  A comparison of different learning-based image fusion methods. The
AE-based method, DenseFuse (Li and Wu, 2018) suffers from a bias issue, by
biasing toward the infrared modality, which leads to the information loss in
the fusion result (yellow boxes). However, as denoted by the red boxes, the
AE-based method can produce more visually pleasing fused images, compared with
the other two paradigms (DDcGAN and MUFusion (Cheng et al., 2023)).

However, according to the investigations in (Ma et al., 2019; Xu et al.,
2020b; Rao et al., 2023), noise and artifacts are also incorporated into the
fusion result, as part of the adversarial learning (pedestrians in Fig. 3
(a)).

In the MEIF field, taking into consideration the structural similarity,
Prabhakar et al. use an autoencoder to integrate the information from
underexposed and overexposed images (Prabhakar et al., 2017). Li and Wu extend
its application to the IVIF task (Li and Wu, 2018) and a series of AE-based
algorithms are proposed in (Li et al., 2020b; Fu and Wu, 2021; Li et al.,
2021). Although the authors devise elaborate fusion rules and even utilize a
trainable network to learn an optimal fusion strategy, the bias issues are
still encountered in these methods, which leads to the loss of information
(the ground in Fig. 3 (c)). On the other hand, for the CNN-based methods
(Zhang et al., 2020a, b; Long et al., 2021; Cheng et al., 2023), they
eliminate the handcrafted feature aggregation processes. However, the loss
functions used in these approaches still rely on the empirical design based on
the information theory (Xu et al., 2020a), activity level maps (Cheng et al.,
2023) or some choose-max strategies (Zhang and Ma, 2021; Tang et al., 2022a),
which share similar risks with the handcrafted fusion rule designs.

In general, CNN-based methods and GAN-based methods usually mix the feature
extraction and feature aggregation processes up. In contrast, in the AE-based
methods, these two stages are separated. Consequently, although the fusion
layer design can sometimes give rise to information loss, the fused images of
the AE-based methods usually are more pleasing, compared with the
aforementioned two paradigms (red boxes of Fig. 3). Considering this merit of
the AE-based approach, we adopt this paradigm in our FusionBooster and propose
a nested AE network to first perceive and then reconstruct the initial fusion
result. As depicted in Fig. 1, by virtue of the FusionBooster, most of the
noise and artifacts contained in the fused images can effectively be
eliminated. Note that, the fusion focus of the backbone method is retained in
the enhanced result, e.g., both the salient thermal information and the rich
texture details are preserved.

Figure 4: The pipeline of the proposed FusionBooster for the MEIF task
(Backbone: U2Fusion). Our booster is composed of three parts, i.e., the
information probe, the booster layer, and the assembling (ASE) module. The
information probe first perceives the source components $I_{\textrm{partA}}$
and $I_{\textrm{partB}}$ in the initial result. The ASE module will piece
these components together to rebuild the initial result. In the test phase,
the degraded components are fine-tuned in the booster layer and the ASE module
correspondingly yields the enhanced result.

###  2.2 Image Fusion Methods with Integrated Enhancement Models

Some of the image fusion methods are derived from the CNN-based approaches.
The significant difference lies in the use of an additional vision model or
some other complementary stages to enhance the fusion performance (Li et al.,
2021; Liu et al., 2022a). Specifically, the methods combined with other vision
tasks train the fusion model and the detection or segmentation model in a
joint or mutually reinforcing manner (Sun et al., 2022; Tang et al., 2022a).
In this way, the performance of both the related vision task and the IVIF task
is expected to benefit. In MetaFusion (Zhao et al., 2023a), Zhao et al.
address the task gap issue of these methods and propose to use a meta-feature
embedding from the detection model to alleviate it. However, their attempt is
not completely satisfactory, as the combination of these features does not
deliver robust fused images (result (b) in Fig. 1).

In contrast, the methods with an additional stage to learn the feature
aggregation process do not suffer from the task gap issue. In RFN-Nest (Li et
al., 2021), Li et al. replace the fusion layer from an AE-based method with a
learnable fusion network. The image fusion task is now transferred into the
feature aggregation task. However, this transformation does not disentangle
the image fusion tasks effectively, as the fusion of the feature maps is as
tricky as the fusion at the pixel level. Consequently, with additional end-to-
end requirements, the quality of the fused images of the RFN-Nest cannot catch
that of the traditional AE-based methods (Cheng et al., 2023).

To address the above-mentioned issues of the existing enhancement-based
methods, we design an information probe module for the image fusion task. The
role of this module is to sense the missing information in the initial fusion
result. In this way, we change the fusion enhancement objective to that of
recovery of the source components by this module, which is more feasible.
Besides, as our FusionBooster does not require a joint training scheme, it can
be even used to enhance the performance of traditional image fusion
approaches. Although no additional vision model is adopted in our booster, the
experimental results demonstrate that, the information preservation strategies
and the sharpening technique used in our booster can also significantly
upgrade the performance of downstream detection tasks.

##  3 The Approach

In this section, we introduce the proposed FusionBooster (FB) architecture in
detail. We assume that the source images for an arbitrary fusion method at
stage one are $I_{\textrm{A}}$ and $I_{\textrm{B}}$. For example, in the MEIF
task, the $I_{\textrm{A}}$ and $I_{\textrm{B}}$ correspond to the underexposed
and overexposed images, respectively. For the backbone method, its initial
fusion result at stage one is denoted as $F_{\textrm{init}}$.

###  3.1 Problem Formulation

In the image fusion field, different fusion tasks pursue the same objective,
which is to preserve information from different modalities or images with
different capture settings. According to this objective, in our approach, we
use the information probe to control the fusion process so as to enhance the
relevant information from the source images and thus boost the performance.

As shown in Fig. 4, our FB follows the divide and conquer strategy, i.e., the
different components of the fusion result are first separated and then
enhanced. Specifically, in the training phase, the information probe learns to
gauge the information conveyed by the source images from the initial result
outputted by the backbone, which is formulated as:

$[I_{\textrm{partA}},I_{\textrm{partB}}]=PU(F_{\textrm{init}}),$ (1)

where $PU$ indicates the probe unit, and $I_{\textrm{partA}}$ and
$I_{\textrm{partB}}$ represent the underexposed and overexposed components,
respectively. With the probe information in hand, the ASE module is tasked to
optimize the assembly of the extracted components to rebuild the initial
fusion result $F_{\textrm{init}}$, i.e.

$\hat{F}_{\textrm{init}}=ASE(I_{\textrm{partA}},I_{\textrm{partB}}),$ (2)

where $\hat{F}_{\textrm{init}}$ denotes the assembly result.

Figure 5: The architecture of the proposed nested AE network. The core of this
model is composed of the probe units and the assembling module. We use the AE-
based architecture to formulate these components. On the other hand, from an
overall (external) perspective, our model can be regarded as an AE to
reconstruct the initial fusion result $F_{\mathrm{init}}$. The encoder and
decoder modules of our network consist of several convolutional layers.
Figure 6: An illustration of the training process of the FusionBooster. The
information probe learns to perceive the source images by utilizing a
perception loss function. The ASE module optimizes the reconstruction loss to
rebuild the initial fusion result.

Given an ideal fusion result, the detached parts in Eq. (1) are expected to
obey the following constraints:

$I_{\textrm{partA}}=I_{\textrm{A}},\ I_{\textrm{partB}}=I_{\textrm{B}}.$ (3)

However, the information loss issues and the artifacts contained in
$F_{\textrm{init}}$ will contaminate these parts and make them degraded. Thus,
in the test phase, we devise a booster layer to recover these two defective
components and improve the assembly result. Since we expect the
$\hat{F}_{\textrm{init}}$ to approximately contain all the information from
the source images (approach the ideal fused image), we set to achieve this
objective in the booster layer by maintaining the upgraded components and
source images as close as possible, i.e.

$\hat{I}_{\textrm{partA}}\approx I_{\textrm{A}},\
\hat{I}_{\textrm{partB}}\approx I_{\textrm{B}},$ (4)

where $\hat{I}_{\textrm{partA}}$ and $\hat{I}_{\textrm{partB}}$ indicate the
boosted components. In this way, the enhanced $F_{\textrm{init}}$ will become
more informative and have refined imaging quality.

Without considering the weight measurement of source images, we only focus on
strengthening the perceived parts of the initial result. Thus, compared to the
conventional approach with one stage being used to handle multiple issues, our
divide and conquer strategy has distinct advantages.

###  3.2  FusionBooster training

The trainable parameters of our FB are from the information probe and the ASE
module. Essentially, our FB only involves reconstruction tasks in the training
process. Thus, as we discussed in Section 2.1, we use the autoencoder(AE)
architecture to implement the ASE module and the probe units. As shown in Fig.
5, our network can be regarded as a nested AE network. Specifically, from the
external point of view, our FusionBooster architecture is reconstructing the
initial result by using the information probe. From the internal view, the
information probe and the ASE module are using three AE networks to divide and
enhance the initial fused image. Here, the encoder and decoder parts of this
network are composed of several convolutional layers.

In Fig. 6, we present the iterative training paradigm of our FusionBooster.
Specifically, we use two loss functions to perceive the source components and
reconstruct the initial fusion result at the pixel level. Thus, the total loss
can be defined as:

$Loss_{\textrm{total}}=Loss_{\textrm{per}}+Loss_{\textrm{rec}},$ (5)

where $Loss_{\textrm{per}}$ and $Loss_{\textrm{rec}}$ indicate the perception
loss and the reconstruction loss.

In the information probe, since we have to handle the diversity of the source
images among different fusion tasks, we assume the perceived images are of
equal importance. Accordingly, the two probe units use the identical network
structure, but their parameters are not shared. The corresponding loss
function is formulated as:

$Loss_{\textrm{per}}=Loss_{\textrm{perA}}+Loss_{\textrm{perB}},$ (6)
$Loss_{\textrm{perA}}=\frac{1}{HW}\sum\limits_{i}\sum\limits_{j}\lvert
I_{\textrm{partA}}(i,j)-I_{A}(i,j)\rvert,$ (7)
$Loss_{\textrm{perB}}=\frac{1}{HW}\sum\limits_{i}\sum\limits_{j}\lvert
I_{\textrm{partB}}(i,j)-I_{B}(i,j)\rvert,$ (8)

where $H$ and $W$ denote the height and width of the images.

On the other hand, the ASE module is responsible for piecing these detached
components together to deliver the reconstructed initial result. We train it
in the second step, keeping all the parameters in the information probe
frozen. The corresponding reconstruction loss function used to optimize this
module is defined as:

$Loss_{\textrm{rec}}=\frac{1}{HW}\sum\limits_{i}\sum\limits_{j}\lvert\hat{F}_{\textrm{init}}(i,j)-F_{\textrm{init}}(i,j)\rvert.$
(9)

Since we do not apply complicated transformations or constrain the detached
components in the feature domain by using the pre-trained model (Long et al.,
2021; Xu et al., 2020a), the ASE module can smoothly rebuild the initial
result and extra computational burden can be avoided.

Figure 7: An illustration of the booster layer. As shown in the highlighted
regions, the decomposed components are unable to recover the information from
the source images perfectly. Based on the supplementary source images and the
image sharpening technique, this layer is designed to enhance these degraded
constituents.

###  3.3 Booster Layer

The booster layer is designed to improve the quality of the fused image.
Simultaneously, it preserves the fusion style of the backbone method, which is
embedded within the detached components. Since we need to cover multiple
fusion tasks, the flexibility would be sacrificed if extra measurements or
parameters were introduced in this layer. Besides, as discussed in Section
3.1, the refined constituent components should approach the source images.
Thus, as shown in Fig. 7, we only use the clean source images $I_{\textrm{A}}$
and $I_{\textrm{B}}$ of different fusion tasks in this layer as the reference
sources. Specifically, for a degraded image component, $e.g.$,
$I_{\textrm{partA}}$, we apply average filtering to obtain the low frequency
component $I_{\textrm{partA}}^{b}$ (base layer) as

$I_{\textrm{partA}}^{b}=I_{\textrm{partA}}*D(k),$ (10)

where $D(k)$ denotes the average filter with the size of $(2k+1)\times(2k+1)$.
Correspondingly, the high-frequency component (the details layer) can be
represented as

$I_{\textrm{partA}}^{d}=I_{\textrm{partA}}-I_{\textrm{partA}}^{b}.$ (11)

The proposed booster layer is expected to take care of the degraded
components. However, we also need to keep the fusion styles or clues in the
output components for the reassembly in the ASE module. Thus, we follow the
image sharpening operation by combining the clean source image with the detail
layer of the degraded component, i.e.

$\hat{I}_{\textrm{partA}}=I_{\textrm{A}}+I_{\textrm{partA}}^{d}.$ (12)

Here, the high-frequency information from the degraded component is expected
to provide fusion clues and edge sharpening for the ASE module. Such
enhancement to the edge information has been demonstrated to be useful for the
downstream tasks Cheng et al. (2023); Liu et al. (2022a). Involving the source
images in the enhanced component $\hat{I}_{\textrm{partA}}$ helps to replace
the degraded base layer with the informative one and forces the ASE module to
deliver a more robust fusion result. The effectiveness of the booster layer
design will be demonstrated in Section 4.5.

##  4 Experiment

###  4.1 Experimental Settings

We apply our FB to three widely investigated image fusion tasks, i.e., the
IVIF task, the MFIF task, and the MEIF task. Three public benchmark datasets
are used in our experiments, including the LLVIP dataset (Jia et al., 2021)
for the IVIF task, MFI-WHU dataset (Zhang et al., 2021) for the MFIF task, and
SCIE dataset (Cai et al., 2018) for the MEIF task.

The LLVIP dataset is very challenging. It is mostly composed of high-quality
infrared and visible image pairs in the low-light environment. The MFI-WHU
dataset contains 120 far-focused and near-focused image pairs of different
scenes. The SCIE dataset consists of 590 high-resolution indoor and outdoor
image sequences with different exposure settings. Considering the small scale
of the last two datasets, we randomly crop $128\times 128$ patches for
augmenting the training data. The number of images or patches used for
training is 12,025, 33,703, and 11,702, respectively. The number of randomly
selected image pairs used for evaluation is 250, 20, and 51, respectively.

Figure 8: Illustration of the qualitative results of the infrared and visible
image fusion on one pair of images from the LLVIP dataset.

This algorithm is implemented in PyTorch and executed on an NVIDIA GeForce RTX
3090 GPU. The Adam optimizer (Kingma and Ba, 2014) is used to update the
parameters of the models with the learning rate of $10^{-4}$. The number of
epochs is set as 10 and the batch size is 2. The filter size $k$ in Eq. (10)
is empirically set as 3. All the competitors’ implementations come from the
code repositories mentioned in the original papers or reproduced by other
researchers.

For the quantitative experiments, five widely used image fusion metrics, i.e.,
visual information fidelity (VIF) (Han et al., 2013), an objective image
fusion performance measure ($Q_{abf}$) (Xydeas et al., 2000), information
entropy (EN) (Roberts et al., 2008), edge intensity (EI) (Xydeas et al.,
2000), and standard deviation (SD) (Cheng et al., 2021) are adopted to
evaluate the fusion performance from different perspectives. Specifically, VIF
measures the distortion between the fusion result and the source images to
indicate the information fidelity. $Q_{abf}$ is used to measure the
preservation ability of the gradient information from the input images. EN and
SD measure the information content and contrast of the image. Finally, the
edge information and clarity of the fusion results are reflected by EI.

Figure 9: Illustration of the qualitative results of the infrared and visible
image fusion on another pair of images from the LLVIP dataset. Table 1: The
quantitative results obtained by the proposed FusionBooster on the LLVIP
dataset, compared with other methods W/O extra model or stage. (Bold: Best;
Underline: Second best)

Method | Venue | SD | EN | VIF | EI | Qabf
---|---|---|---|---|---|---
GTF | 16’ Inf. Fus. | 50.164 | 7.351 | 0.576 | 44.129 | 0.454
U2Fusion | 20’ TPAMI | 37.428 | 6.707 | 0.492 | 46.899 | 0.499
DDcGAN | 20’ TIP | 51.495 | 7.431 | 0.764 | 49.127 | 0.395
SD-Net | 21’ IJCV | 36.257 | 6.889 | 0.414 | 44.609 | 0.482
ReCoNet | 22’ ECCV | 48.761 | 5.962 | 0.727 | 47.178 | 0.462
TarDAL | 22’ CVPR | 52.106 | 7.353 | 0.809 | 46.126 | 0.444
YDTR | 22’ TMM | 36.502 | 6.782 | 0.388 | 31.336 | 0.334
LRRNet | 23’ TPAMI | 29.826 | 6.423 | 0.342 | 34.928 | 0.406
MUFusion | 23’ Inf. Fus. | 40.104 | 7.019 | 0.755 | 57.698 | 0.547
YDTR (FB) | Ours | 41.159 | 6.988 | 0.532 | 48.590 | 0.470
GTF (FB) | Ours | 53.260 | 7.412 | 0.884 | 73.185 | 0.485
DDcGAN (FB) | Ours | 57.672 | 7.650 | 0.986 | 75.362 | 0.470

Table 2: The quantitative results obtained by the proposed FusionBooster on
the LLVIP dataset, compared with other methods W/ extra model or stage. (TS:
Two-stage; OD: Object detection; Seg: Segmentation; LE: Low-light enhancement;
FB: FusionBooster)

Method | Venue | Model/Stage (MB) | SD | EN | VIF | EI | Qabf
---|---|---|---|---|---|---|---
RFN-Nest | 21’ Inf. Fus. | TS (17.179) | 39.719 | 7.064 | 0.466 | 34.195 | 0.384
TarDAL++ | 22’ CVPR | OD (14.46) | 41.059 | 6.604 | 0.676 | 70.005 | 0.367
SeAFusion | 22’ Inf. Fus | Seg (0.646) | 51.810 | 7.451 | 0.839 | 55.935 | 0.618
DIVFusion | 23’ Inf. Fus | LE (225.668) | 53.370 | 7.556 | 1.234 | 56.595 | 0.349
CDDFuse | 23’ CVPR | TS (1.462) | 50.409 | 7.374 | 0.787 | 52.324 | 0.622
MetaFusion | 23’ CVPR | OD (14.46) | 49.935 | 7.148 | 1.539 | 81.840 | 0.436
DDcGAN (FB) | Ours | FB (0.168) | 57.672 | 7.650 | 0.986 | 75.362 | 0.470

###  4.2 An Infrared and Visible Image Fusion Task

In this section, we present the fusion results obtained by advanced image
fusion methods and some algorithms enhanced by our booster. As it is an
important task in the image fusion field, we select more competitor algorithms
for the comparative evaluation. The tested algorithms include the traditional
method GTF (Ma et al., 2016), 5 CNN-based methods, namely U2Fusion (Xu et al.,
2020a), SDNet (Zhang and Ma, 2021), ReCoNet Huang et al. (2022), LRRNet Li et
al. (2023) and MUFusion (Cheng et al., 2023), 6 approaches that contain
additioanl enhancement model or fusion stage, $i.e.$, RFN-Nest Li et al.
(2021), TarDAL++ Liu et al. (2022a), SeAFusion (Tang et al., 2022a), DIVFusion
Tang et al. (2023a), CDDFuse Zhao et al. (2023b), and MetaFusion Zhao et al.
(2023a), the GAN-based method DDcGAN (Ma et al., 2020a), and the transformer-
based method YDTR (Tang et al., 2022c).

####  4.2.1 Qualitative Experiments

For the IVIF task, due to the limitations of the handcrafted image features,
the traditional methods cannot handle complex scenes effectively. As shown in
Fig. 8 and Fig. 9, the traditional method, GTF, suffers from the blurring
issues in the fusion results. Our booster can effectively address this and
produce visually pleasing images. Meanwhile, our paradigm also reduces the
artifacts, which severely degrade the image quality of DDcGAN. Besides,
compared with the SOTA methods LRRNet and TarDAL, the enhanced DDcGAN inherits
the merits of the original method and shows the ability to cope with the
challenges of dark environments, preserving the details of the background
(blue boxes), and presenting more salient thermal information on the
foregrounds. Finally, when the object detection model is used to enhance the
TarDAL, compared with the original method, the fusion results of this method
show a lack of brightness in the background and the thermal radiation in the
target regions, which is consistent with our discussion about the task gap
issue. In Section 4.6.1, we further demonstrate the impact of our booster on
TarDAL. The results indicate that our approach is able to mitigate this issue.

Figure 10: Visualization of the results obtained by DDcGAN and DDcGAN with
FusionBooster on the pedestrian detection task.

####  4.2.2 Quantitative Experiments

For the quantitative comparison, we select three different types of fusion
methods, i.e., the traditional method GTF, the transformer-based method YDTR,
and the GAN-based method DDcGAN as the backbone methods of our booster.
Meanwhile, for the competitors, we also divide them into two categories, i.e.,
methods with or without using an extra model or stage. As shown in Table 1,
our booster consistently improves the performance of various types of
algorithms on all of these five metrics. The remarkable performance on these
metrics indicates that the proposed booster is able to increase the fidelity
of the information derived from the source images (VIF), better preserve the
gradient information (Qabf), and produce robust fused images with sharp edge
information (EN, SD, and EI). Moreover, the DDcGAN, proposed in 2020 and
upgraded by our FusionBooster, outperforms current SOTA methods in 4 out of 5
metrics, which is a significant improvement.

In addition, we use the upgraded DDcGAN to conduct further experiments
involving other methods, with an extra stage or enhancement module. As shown
in Table 2, our FusionBooster has the smallest volume compared to other
enhancement models. For the quantitative results, the MetaFusion exhibits a
similar performance as our upgraded method. However, as shown in Fig. 8 and
Fig. 9, its inability to address the task gap issue results in poor
performance on the metrics of SD and EN. By contrast, our fusion results
effectively achieve a balance between the image quality, and the correlation
with the source images, obtaining the best performance on the non-reference
metrics and comparable results on the VIF and Qabf. The best overall
performance in this comparison also demonstrates that there is a scope for the
existing image fusion methods to achieve performance gains without considering
other vision tasks.

Table 3: The accuracy of pedestrian detection using different modalities on
the LLVIP dataset.

Method | Venue |  $mAP_{50:95}$(%) |  $mAP_{50}$(%)
---|---|---|---
Visible | Input | 54.2 | 94.4
DDcGAN | 20’ TIP | 63.7 | 94.4
DIVFusion (LE) | 23’ Inf. Fus. | 64.2 | 97.1
YDTR | 22’ TMM | 64.9 | 97.4
CDDFuse (TS) | 23’ CVPR | 65.4 | 97.0
GTF | 16’ Inf. Fus | 65.7 | 96.5
MetaFusion (OD) | 23’ CVPR | 66.1 | 97.3
MUFusion | 23’ Inf. Fus. | 66.4 | 96.2
DenseFuse | 18’ TIP | 66.5 | 96.4
LRRNet | 23’ TPAMI | 66.5 | 97.0
TarDAL | 22’ CVPR | 66.6 | 96.9
SeAFusion (Seg) | 22’ Inf. Fus. | 66.9 | 97.2
U2Fusion | 20’ TPAMI | 67.3 | 97.0
Infrared | Input | 67.9 | 97.3
SDNet | 21’ IJCV | 68.1 | 97.3
TarDAL++ (OD) | 22’ CVPR | 68.3 | 97.2
GTF (FB) | Ours | 67.8 (+2.1) | 97.0 (+0.5)
YDTR (FB) | Ours | 67.6 (+2.7) | 97.9 (+0.5)
DDcGAN (FB) | Ours | 69.3 (+5.6) | 97.4 (+3.0)

Figure 11: The qualitative results achieved in the MFIF task on two pairs of
images from the MFI-WHU dataset.

####  4.2.3 The Pedestrian Detection Task

In addition to the visual quality, an important application for the IVIF task
is to improve the performance of downstream vision tasks by using the
complementary information contained in the fused image. In this experiment, we
use the YOLOv5 detector to test the accuracy of different image fusion methods
on the pedestrian detection task. We separately train the detector by using
the fusion results of different algorithms on the training set of the LLVIP
dataset. The trained models are used to detect pedestrians in different
modalities. As shown in Table 3, in the low-light environment, the accuracy of
some SOTA methods cannot even match that of the single modality, i.e., the
infrared modality. However, once the FB is applied in conjunction with these
methods, the average precision is significantly improved, e.g., $5.6\%$ for
DDcGAN over the IoU thresholds from 0.5 to 0.95. It is worth noting that the
performance of DDcGAN with our booster is better than that of the SeAFusion
and TarDAL++, which consider similar segmentation and detection tasks in their
training process.

In Fig. 10, we present the visualization of two results obtained with our
booster in the pedestrian detection task. The detector has a higher confidence
for the detected pedestrians and the false detection issues are mitigated
(bike in the top left corner of the first example). This comparison also
reveals that the fusion results with sharpened edge information and higher
contrast can benefit the detection task, which is consistent with the
motivation of designing the booster layer mentioned in Section 3.3.

Figure 12: The quantitative results obtained by different fusion methods with
(bold) and without the FB in the case of the MFIF task. Figure 13: The
qualitative results, obtained on two pairs of images from the SCIE dataset,
when performing the MEIF task. Table 4: The quantitative results, obtained on
two pairs of images from the SCIE dataset, when performing the MEIF task.

Method | Venue | SD | EN | VIF | EI | Qabf
---|---|---|---|---|---|---
DeepFuse | 17’ ICCV | 46.091 | 7.155 | 1.295 | 60.526 | 0.696
TLER | 18’ SPL | 41.443 | 7.103 | 1.713 | 75.570 | 0.736
MEF-GAN | 20’ TIP | 52.363 | 7.192 | 1.592 | 80.214 | 0.373
U2Fusion | 20’ TPAMI | 49.013 | 7.213 | 1.695 | 83.001 | 0.638
SDNet | 21’ IJCV | 44.135 | 7.035 | 1.299 | 72.497 | 0.677
AGAL | 22’ TCSVT | 43.725 | 7.106 | 1.314 | 71.954 | 0.655
MUFusion | 23’ Inf. Fus. | 49.682 | 7.231 | 1.637 | 70.179 | 0.716
IID-MEF | 23’ Inf. Fus. | 40.975 | 7.035 | 1.124 | 59.117 | 0.610
TLER (FB) | Ours | 50.187 | 7.249 | 1.934 | 110.503 | 0.518
U2Fusion (FB) | Ours | 58.573 | 7.506 | 2.506 | 134.524 | 0.425

###  4.3 Multi-focus Image Fusion

In this section, we present the MFIF results obtained by various image fusion
methods and two boosting examples of our booster. For this task, we select 4
methods, i.e., PMGI (Zhang et al., 2020a), DRPL (Li et al., 2020d), UNIFusion
(Cheng et al., 2021), and ZMFF Hu et al. (2023) as the competitors.

####  4.3.1 Qualitative Experiments

Due to the general operations used in the booster layer, our FusionBooster is
also able to improve existing multi-focus image fusion methods. As shown in
the first row of Fig. 11, applying the proposed booster to the traditional
method, CSR, and the learning-based method, U2Fusion, the details on the board
of the bus become clearer. Furthermore, in the second example, the original
CSR does not accurately infer the focused regions of the source images (head
of the “giraffe’). As shown in the magnified region, the enhanced result
successfully addresses this issue by improving the clarity. Similar conclusion
can be reached by looking at the enhancement achieved by U2Fusion. In
conclusion, compared with the other algorithms, the enhanced methods are
superior in terms of preserving the local details in the highlighted area.

####  4.3.2 Quantitative Experiments

For the quantitative results, as shown in Fig. 12, with our booster (legends
with bold borders), U2Fusion has a clear advantage over the other advanced
methods in terms of VIF, EI, EN, and SD. This promising result demonstrates
the superiority of the proposed FusionBooster. Moreover, integrating with our
booster, the traditional method CSR (Liu et al., 2016) also exhibits distinct
strengths on multiple metrics. However, our FB does not perform well on the
metric of Qabf. Similar issues also arise in the context of related work Li et
al. (2020c); Cheng et al. (2023). This can be attributed to the enhancement
effect of our FusionBooster, which alters the gradient information transferred
from the source images into the fusion results. Consequently, this gradient
based metric cannot consistently reflect the benefits of our booster.

###  4.4 Multi-exposure Image Fusion Task

In this section, we present the MEIF results obtained by different methods and
some algorithms upgraded with our booster. Four open source CNN-based MEIF
algorithms, i.e., DeepFuse (Prabhakar et al., 2017), MEF-GAN (Xu et al.,
2020b), AGAL (Liu et al., 2022b), and IID-MEF (Zhang and Ma, 2023), and a
traditional method TLER (Yang et al., 2018) are involved in the experiments.

Figure 14: Qualitative results of the ablation experiments. Without using the
second stage, the explicit enhancement in the first stage (settings (a) and
(b)) makes different fusion results blurred in the highlighted regions.
Meanwhile, without the detached components from the information probe (setting
(c) and (d)), the edge information in the IVIF results is not clear and there
are some artifacts in the MEIF results. The yellow regions in the MEIF results
denote the exposure difference in various experimental settings.  Figure 15:
Illustration of different ablation experiments investigating the enhancement
operation. The enhancement operator corresponds to the sharpening operation in
the booster layer. (a): Enhancing the input for the backbone method; (b):
Directly enhancing the backbone method’s fusion result; (c): Without the
information probe, the input for the ASE module is the source images; (d):
Without the information probe, the input for the ASE module is the enhanced
source images.

####  4.4.1 Qualitative Experiments

As a gradient-based image fusion method, U2Fusion delivers promising results
in other fusion tasks. In the case of the MEIF task, as shown in Fig. 13, its
gradient-based information measurement ignores the adaptation of the exposure
setting and the results tend to preserve more information from the
underexposed image. This observation indicates that it is tricky to consider
all aspects of the fusion process in a single stage. By contrast, our booster
effectively mitigates this issue by lighting the dark area of the original
results in the second stage. Meanwhile, as shown in the magnified regions, our
booster also enhances the edge information and generates results of higher
clarity. When compared with other SOTA methods (SDNet and MUFusion), the
enhanced fusion results benefit from maintaining an appropriate level of
exposure and succeed in preserving abundant texture details.

####  4.4.2 Quantitative Experiments

We also compare the performance of different MEIF methods in the image quality
assessments. As shown in Table 4, the enhanced traditional and learning-based
(U2Fusion) methods achieve consistent improvements in these metrics, as in the
previous MFIF task. Note that the best performance in most of the metrics over
other SOTA methods in multiple fusion tasks demonstrates the powerful
generalization capability of our concise booster design.

Table 5: Quantitative results of the ablation experiments in three different
fusion tasks.

Methods | IVIF (Backbone: GTF) | MFIF (Backbone: CSR) | MEIF (Backbone: U2Fusion)
---|---|---|---
SD | EN | VIF | EI | Qabf | SD | EN | VIF | EI | Qabf | SD | EN | VIF | EI | Qabf
Baseline | 50.16 | 7.35 | 0.58 | 44.13 | 0.45 | 54.09 | 7.16 | 1.09 | 75.04 | 0.72 | 49.01 | 7.21 | 1.69 | 83.00 | 0.64
Baseline + (a) | 28.99 | 6.64 | 0.23 | 33.25 | 0.33 | 31.21 | 6.59 | 0.50 | 68.26 | 0.62 | 30.30 | 6.70 | 0.83 | 74.14 | 0.53
Baseline + (b) | 24.35 | 6.40 | 0.18 | 31.51 | 0.30 | 29.19 | 6.44 | 0.46 | 65.88 | 0.62 | 27.50 | 6.50 | 0.69 | 65.99 | 0.53
Baseline + (c) | 50.97 | 7.37 | 0.60 | 46.68 | 0.41 | 58.91 | 7.26 | 1.28 | 78.47 | 0.67 | 51.39 | 7.27 | 2.14 | 113.03 | 0.54
Baseline + (d) | 51.79 | 7.39 | 0.72 | 70.74 | 0.41 | 60.74 | 7.37 | 1.75 | 126.89 | 0.47 | 52.07 | 7.39 | 2.29 | 147.93 | 0.39
Baseline (FB) | 53.26 | 7.41 | 0.88 | 73.19 | 0.48 | 61.17 | 7.33 | 1.79 | 127.01 | 0.45 | 58.57 | 7.51 | 2.51 | 134.52 | 0.43

###  4.5 Ablation Experiments

####  4.5.1 The Impact of the Information Probe and the Booster Layer

In this section, we present more ablation experiments evaluating our booster
on different image fusion tasks. An illustration of our experimental settings
is presented in Fig. 15.

Firstly, we validate the need to deploy the second stage to enhance the fusion
results. In setting (a), we enhance the source images for the backbone method
to make it produce more promising results. In setting (b), we directly enhance
the fusion result of the backbone method. On the other hand, we want to
validate the effectiveness of the proposed information probe and the
corresponding enhancement procedure used in the booster layer. In settings (c)
and (d), we discard the information probe and only use the source images
(setting (c)) and enhanced source images (setting (d)) as the input of the ASE
module, respectively. These four experiments are all conducted independently.
Here, the enhancement relates to the sharpening operation in the booster layer
(the detail layer is from the image itself).

As shown in Table 5, our strategy in the booster layer enables the backbone
methods to have the best overall performance in these three fusion tasks.
Specifically, the setting in our FusionBooster performs better in the IVIF
task (5 best metrics). As shown in Fig. 14, directly enhancing the source
images or fusion results in the first stage will make the IVIF results blurred
to a certain extent, and the bad exposure settings in the MEIF results cannot
be improved (settings (a) and (b)). Without using the information probe to
divide different components apart (settings (c) and (d)), the enhancement
quality cannot be guaranteed, e.g., the edge information from the IVIF result
is not clear and the enhanced results from the MEIF task tend to produce some
artifacts.

Figure 16: The perception results of the information probe otained in three
image fusion tasks. As shown in the highlighted regions of the MFIF results,
compared with the other two perception results, the information probe does not
produce output with a focus on different areas of the source images. Instead,
it only produces images with all-focused or all-blurred styles.  Figure 17:
Reconstruction examples of the information probe oftained for two
representative methods TarDAL and DDcGAN. As denoted by the red boxes, the
degradation of the reconstructed images is related to the fusion performance
of the initial results.

####  4.5.2 An Analysis of the Information Probe

The performance gain of our booster on the MFIF task is not as significant as
it is for the other two tasks, i.e., the differences between the setting (d)
and our booster are not very distinct in the visualization results. We carried
out further experiments to investigate this issue. In Fig. 16, we show the
perception results of our information probe in the case of these three fusion
tasks. In the IVIF task and the MEIF task, our booster can coarsely recover
the complementary source images. However, in the MFIF task, lacking the
necessary depth information, our probe can only produce images with either
all-focused or all-blurred styles, which are not consistent with the source
images. In this particular example, the limited overlap between the
disentangled components and the supplementary source images leads to
suboptimal enhancement effects.

The key aim of our FB is to enhance the components outputted by the
information probe to improve the fusion performance. The success of this
operation is related to degradations caused by the information probe.
Specifically, inaccurate estimates of weighting, and any artifacts injected by
the backbone will make it difficult to reconstruct the source images from the
initial fusion result. In these circumstances the separated components will
tend to degrade. We argue that the quality of the fusion results is correlated
with the degree of degradation of the reconstructed images. Thus, if we
recover these components successfully, and then combine them, theoretically,
we should obtain better fused images.

From previous experiments, we notice that the TarDAL is able to produce fused
images with a high-quality visual effect, while the DDcGAN generates some
noise and artifacts in their fusion results. In Fig. 17, we use these two
methods to conduct experiments investigating the impact of degradation. As
shown in the red boxes, with higher image quality, the extent of degradation
produced by TarDAL is less than that of the DDcGAN. For example, the thermal
radiation looks clearer in the infrared component of TarDAL. Meanwhile, in the
visible spectrum, the DDcGAN cannot recover the texture details from the image
as effectively as TarDAL. These observations are consistent with our
expectations of the effect of degradation caused by the information probe.

Figure 18: More results of the proposed FusionBooster combined with other
advanced IVIF algorithms. As denoted in the highlighted regions, integrating
SDNet with our booster preserves the detail information from the background
better. The salience of the targets is increased in the results of TarDAL, and
the artifacts contained in the MUFusion are eliminated. ( Blue: Better
performance obtained using FusionBooster )  Figure 19: More qualitative
results of the proposed FusionBooster integrated with other MFIF algorithms.
As denoted in the magnified areas, our booster, integrated with the advanced
fusion methods MUFusion and DRP, presents clearer texture details. The
original PMGI produces blurring. Since the high-quality supplement source is
used in the booster layer, our FusionBooster effectively mitigates this issue.
Figure 20: More qualitative results obtained by the proposed FusionBooster in
conjunction with with other MEIF algorithms. As shown in the highlighted
regions, SDNet obtains better exposure in the dark regions thanks to
FusionBooster. The unnatural colour and artifacts visible in the fusion
results of AGAL and GFF are also effectively removed in the refined output.
Figure 21: The quantitative results of the proposed FB combined with more MFIF
algorithms. Generally, our FB can consistently enhance the performance of most
image fusion methods. However, the gain from the FB can be very different for
various backbone methods (the performance of Qabf and EN).  Table 6: The
quantitative results obtained by the proposed FusionBooster (FB) in
conjunction with more MEIF algorithms. ( Bold: Better performance obtained
using FusionBooster )

Method | SD | EN | VIF | Qabf | EI
---|---|---|---|---|---
GFF | 47.072 | 7.382 | 1.527 | 0.653 | 84.293
+FB | 53.557 | 7.132 | 2.002 | 0.514 | 108.172
DeppFuse | 46.091 | 7.156 | 1.295 | 0.696 | 60.527
+FB | 45.918 | 7.169 | 1.320 | 0.590 | 80.146
SDNet | 44.135 | 7.035 | 1.299 | 0.677 | 72.497
+FB | 43.890 | 7.183 | 1.408 | 0.521 | 101.926
AGAL | 43.725 | 7.106 | 1.314 | 0.655 | 71.954
+FB | 44.930 | 7.208 | 1.383 | 0.507 | 97.936
MUFusion | 49.682 | 7.231 | 1.637 | 0.716 | 70.179
+FB | 49.836 | 7.293 | 1.694 | 0.561 | 96.542

###  4.6 More Results in Different Fusion Tasks

####  4.6.1 More Results in the IVIF Task

In this section, we present more experimental results of our booster
integrated with other IVIF algorithms. In this experiment, both day and night
scenes are covered in our test images. As shown in Fig. 18, our booster
consistently strengthens the performance of these algorithms in the following
sense: the preservation of more texture details in the background regions
(SDNet), the improvement in the capture of salience of the thermal radiation
(TarDAL++), and the reduction of artifacts (MUFusion). These advanced methods
all achieve better quantitative performance when combined with our booster in
different image quality assessments, e.g., huge increase in the performance of
VIF and EI, which demonstrates the superiority of our booster in the
performance of the IVIF task. In some of the cases, e.g., the fusion results
of MUFusion(FB), our booster enhances the performance of the quantitative
results only marginally. However, the benefit of reducing artifacts should not
be underestimated, which is not well reflected by the adopted metrics.

####  4.6.2 More Results in the MFIF Task

We conducted further experiments relating to the MFIF task to validate the
effectiveness of the proposed FusionBooster. As shown in the highlighted
regions of Fig. 19, the blurring issue in the fusion result of PMGI Zhang et
al. (2020a) is effectively mitigated by our booster. The proposed FB also
enhances the preserved detail information of other fusion approaches, e.g.,
the fence in front of the door is clearer in the refined images. In Fig. 21,
we also evaluate the quantitative performance of different image fusion
methods and their upgraded versions. Our FusionBooster effectively boosts the
performance of different fusion methods on most of these five metrics, e.g.,
around 33% improvement in the metric of visual information fidelity for the
MUFusion. Although the PMGI has worse performance in the metric of EN, the
higher quality images yielded in the qualitative experiments indicate that our
booster works well in conjunction with this algorithm. Besides, the much more
abundant texture details from the improved PMGI is also consistent with the
significant increase in the metric of EI.

####  4.6.3 More Results in the MEIF Task

In this section, we conduct experiments to evaluate FusionBooster integrated
with more MEIF algorithms. As shown in Fig. 20, applying our booster to the
SDNet Zhang and Ma (2021) results in a more appropriate exposure in the dark
regions of the original fused images. Note also, as presented in the magnified
regions, the original results of the AGAL Liu et al. (2022b) and the
traditional approach GFF Li et al. (2013) are not satisfactory. Specifically,
they exhibit unnatural colour and artifacts in the output images. The refined
results appear to address these issues effectively.

Finally, for the quantitative experiments (Table 6), the reasons for poor
performance on the metric of Qabf have been explained in Section. 4.3.2. In
consistency with the conclusions drawn from the previous experiments, our
booster enables all MEIF methods to achieve significant improvements in most
of the image quality assessments.

Table 7: The inference time and the model size comparison of different methods
on 250 image pairs from the LLVIP dataset. (Bold: extra cost)

Metric | U2Fusion | MUFusion | YDTR | SeAFusion | DenseFuse | GTF | GTF + FB | DDcGAN | DDcGAN + FB | SDNet | SDNet + FB
---|---|---|---|---|---|---|---|---|---|---|---
Time (s) | 66.90 | 52.98 | 28.01 | 7.51 | 1.82 | 128.67 | 130.66 (+1.99) | 123.12 | 125.04 (+1.92) | 2.14 | 4.13 (+1.99)
Model size (MB) | 2.51 | 2.12 | 0.85 | 0.65 | 0.29 | – | – | 21.18 | 21.35 (+0.17) | 0.26 | 0.43 (+0.17)

###  4.7 Comparison of the Computational Complexity and Model Size

In this section, we provide the statistics of additional time consumption and
model size burden for several image fusion methods utilizing the proposed
FusionBooster. The information is presented in Table 7, where we collect the
inference time of several approaches, as well as their model sizes in the
context of the IVIF task on the LLVIP dataset. While achieving much better
performance in various fusion tasks, our booster increases the time
consumption of the baseline methods by only around 2 seconds on 250 infrared
and visible image pairs, and increases the size of the model by less than
200KB. Such lightweight solution offers attractive advantages in comparison
with the expensive computational cost issue of the existing enhancement-based
methods.

##  5 Conclusion

In this paper, we proposed an image fusion enhancer based on a divide and
conquer strategy guided by an innovative information probe. It is the first
time such a universally applicable boosting paradigm is proposed in the
literature. Given a fused image from an arbitrary method, e.g., an IVIF
algorithm, we first decompose the initial result into different components.
The information probe gauges the affinity of the components to the input
images, and filters them to yield an improved fused image. The difference
signal iteratively drives the update of the FusionBoster parameters. In this
way, we effectively mitigate the information loss and image blurring issues in
the backbone. The nested AE design of the network architecture and the loss
function are the key ingredients of the improved performance at the expense of
a minor increase in the computational cost. Compared with other extra modules
required by the enhancement-based methods, the proposed booster can be applied
to different fusion tasks very effectively. Moreover, it significantly boosts
various fusion approaches, including the traditional and learning-based
methods.

Although our booster significantly enhances existing image fusion methods, it
leaves some scope for future research. Firstly, in our FB, we study the image
fusion enhancement only from the perspective of information retention. As
presented in some of the experiments, our FB cannot always improve the image
fusion performance. Thus, investigating diverse manners to disentangle and
analyse the fused images may probably benefit the performance of the booster.
Secondly, the effective enhancement strategy delivered by the booster layer
could potentially be further improved in the future by a trainable booster
network. Finally, the proposed booster has only been validated in a limited
number of fusion tasks. The generalization ability of this approach remains to
be proven in other applications.

## Availability of Data and Materials

Information on access to the datasets supporting the conclusions of this
article is included therein.

## Competing Interests

The authors declare that they have no conflict of interest.

###### Acknowledgements.

This work was supported by the National Natural Science Foundation of China
(62020106012, U1836218, 62106089, 62202205), the 111 Project of Ministry of
Education of China (B12018), the Engineering and Physical Sciences Research
Council (EPSRC) (EP/R018456/1, EP/V002856/1), and the Postgraduate Research &
Practice Innovation Program of Jiangsu Province (KYCX23_2525).

## References

  * Cai et al. (2018) Cai J, Gu S, Zhang L (2018) Learning a deep single image contrast enhancer from multi-exposure images. IEEE Transactions on Image Processing 27(4):2049–2062
  * Chen et al. (2021) Chen J, Li X, Luo L, Ma J (2021) Multi-focus image fusion based on multi-scale gradients and image matting. IEEE Transactions on Multimedia 24:655–667
  * Cheng et al. (2021) Cheng C, Wu XJ, Xu T, Chen G (2021) Unifusion: A lightweight unified image fusion network. IEEE Transactions on Instrumentation and Measurement 70:1–14
  * Cheng et al. (2023) Cheng C, Xu T, Wu XJ (2023) Mufusion: A general unsupervised image fusion network based on memory unit. Information Fusion 92:80–92
  * Fu and Wu (2021) Fu Y, Wu XJ (2021) A dual-branch network for infrared and visible image fusion. In: 2020 25th International Conference on Pattern Recognition (ICPR), IEEE, pp 10675–10680
  * Fu et al. (2021) Fu Y, Wu XJ, Durrani T (2021) Image fusion based on generative adversarial network consistent with perception. Information Fusion 72:110–125
  * Han et al. (2013) Han Y, Cai Y, Cao Y, Xu X (2013) A new image fusion performance metric based on visual information fidelity. Information fusion 14(2):127–135
  * Hu et al. (2023) Hu X, Jiang J, Liu X, Ma J (2023) Zmff: Zero-shot multi-focus image fusion. Information Fusion 92:127–138
  * Huang et al. (2022) Huang Z, Liu J, Fan X, Liu R, Zhong W, Luo Z (2022) Reconet: Recurrent correction network for fast and efficient multi-modality image fusion. In: European Conference on Computer Vision, Springer, pp 539–555
  * Jia et al. (2021) Jia X, Zhu C, Li M, Tang W, Zhou W (2021) Llvip: A visible-infrared paired dataset for low-light vision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 3496–3504
  * Kingma and Ba (2014) Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980
  * Li and Wu (2018) Li H, Wu XJ (2018) Densefuse: A fusion approach to infrared and visible images. IEEE Transactions on Image Processing 28(5):2614–2623
  * Li et al. (2020a) Li H, Ma K, Yong H, Zhang L (2020a) Fast multi-scale structural patch decomposition for multi-exposure image fusion. IEEE Transactions on Image Processing 29:5805–5816
  * Li et al. (2020b) Li H, Wu XJ, Durrani T (2020b) NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models. IEEE Transactions on Instrumentation and Measurement 69(12):9645–9656
  * Li et al. (2020c) Li H, Wu XJ, Kittler J (2020c) Mdlatlrr: A novel decomposition method for infrared and visible image fusion. IEEE Transactions on Image Processing 29:4733–4746
  * Li et al. (2021) Li H, Wu XJ, Kittler J (2021) Rfn-nest: An end-to-end residual fusion network for infrared and visible images. Information Fusion 73:72–86
  * Li et al. (2023) Li H, Xu T, Wu XJ, Lu J, Kittler J (2023) Lrrnet: A novel representation learning guided fusion network for infrared and visible images. IEEE transactions on pattern analysis and machine intelligence
  * Li et al. (2020d) Li J, Guo X, Lu G, Zhang B, Xu Y, Wu F, Zhang D (2020d) Drpl: Deep regression pair learning for multi-focus image fusion. IEEE Transactions on Image Processing 29:4816–4831
  * Li et al. (2013) Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Transactions on Image processing 22(7):2864–2875
  * Liu et al. (2022a) Liu J, Fan X, Huang Z, Wu G, Liu R, Zhong W, Luo Z (2022a) Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5802–5811
  * Liu et al. (2022b) Liu J, Shang J, Liu R, Fan X (2022b) Attention-guided global-local adversarial learning for detail-preserving multi-exposure image fusion. IEEE Transactions on Circuits and Systems for Video Technology
  * Liu et al. (2016) Liu Y, Chen X, Ward RK, Wang ZJ (2016) Image fusion with convolutional sparse representation. IEEE signal processing letters 23(12):1882–1886
  * Long et al. (2021) Long Y, Jia H, Zhong Y, Jiang Y, Jia Y (2021) Rxdnfuse: A aggregated residual dense network for infrared and visible image fusion. Information Fusion 69:128–141
  * Ma et al. (2016) Ma J, Chen C, Li C, Huang J (2016) Infrared and visible image fusion via gradient transfer and total variation minimization. Information Fusion 31:100–109
  * Ma et al. (2019) Ma J, Yu W, Liang P, Li C, Jiang J (2019) Fusiongan: A generative adversarial network for infrared and visible image fusion. Information Fusion 48:11–26
  * Ma et al. (2020a) Ma J, Xu H, Jiang J, Mei X, Zhang XP (2020a) Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Transactions on Image Processing 29:4980–4995
  * Ma et al. (2020b) Ma J, Zhang H, Shao Z, Liang P, Xu H (2020b) Ganmcc: A generative adversarial network with multiclassification constraints for infrared and visible image fusion. IEEE Transactions on Instrumentation and Measurement 70:1–14
  * Prabhakar et al. (2017) Prabhakar KR, Srikar VS, Babu RV (2017) Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: ICCV, vol 1, p 3
  * Rao et al. (2023) Rao D, Xu T, Wu XJ (2023) Tgfuse: An infrared and visible image fusion approach based on transformer and generative adversarial network. IEEE Transactions on Image Processing
  * Roberts et al. (2008) Roberts JW, Van Aardt JA, Ahmed FB (2008) Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing 2(1):023522
  * Sun et al. (2022) Sun Y, Cao B, Zhu P, Hu Q (2022) Detfusion: A detection-driven infrared and visible image fusion network. In: Proceedings of the 30th ACM International Conference on Multimedia, pp 4003–4011
  * Tang et al. (2022a) Tang L, Yuan J, Ma J (2022a) Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Information Fusion 82:28–42
  * Tang et al. (2022b) Tang L, Yuan J, Zhang H, Jiang X, Ma J (2022b) Piafusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion
  * Tang et al. (2023a) Tang L, Xiang X, Zhang H, Gong M, Ma J (2023a) Divfusion: Darkness-free infrared and visible image fusion. Information Fusion 91:477–493
  * Tang et al. (2022c) Tang W, He F, Liu Y (2022c) Ydtr: infrared and visible image fusion via y-shape dynamic transformer. IEEE Transactions on Multimedia
  * Tang et al. (2023b) Tang Z, Xu T, Li H, Wu XJ, Zhu X, Kittler J (2023b) Exploring fusion strategies for accurate rgbt visual object tracking. Information Fusion p 101881
  * Xu et al. (2022a) Xu F, Liu J, Song Y, Sun H, Wang X (2022a) Multi-exposure image fusion techniques: A comprehensive review. Remote Sensing 14(3):771
  * Xu et al. (2020a) Xu H, Ma J, Jiang J, Guo X, Ling H (2020a) U2fusion: A unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence
  * Xu et al. (2020b) Xu H, Ma J, Zhang XP (2020b) Mef-gan: multi-exposure image fusion via generative adversarial networks. IEEE Transactions on Image Processing 29:7203–7216
  * Xu et al. (2022b) Xu H, Ma J, Yuan J, Le Z, Liu W (2022b) Rfnet: Unsupervised network for mutually reinforcing multi-modal image registration and fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 19679–19688
  * Xu et al. (2019) Xu T, Feng ZH, Wu XJ, Kittler J (2019) Joint group feature selection and discriminative filter learning for robust visual object tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 7950–7960
  * Xydeas et al. (2000) Xydeas CS, Petrovic V, et al. (2000) Objective image fusion performance measure. Electronics letters 36(4):308–309
  * Yang et al. (2018) Yang Y, Cao W, Wu S, Li Z (2018) Multi-scale fusion of two large-exposure-ratio images. IEEE Signal Processing Letters 25(12):1885–1889
  * Zhang and Ma (2021) Zhang H, Ma J (2021) Sdnet: A versatile squeeze-and-decomposition network for real-time image fusion. International Journal of Computer Vision pp 1–25
  * Zhang and Ma (2023) Zhang H, Ma J (2023) Iid-mef: A multi-exposure fusion network based on intrinsic image decomposition. Information Fusion 95:326–340
  * Zhang et al. (2020a) Zhang H, Xu H, Xiao Y, Guo X, Ma J (2020a) Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, pp 12797–12804
  * Zhang et al. (2021) Zhang H, Le Z, Shao Z, Xu H, Ma J (2021) Mff-gan: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Information Fusion 66:40–53
  * Zhang (2021) Zhang X (2021) Deep learning-based multi-focus image fusion: A survey and a comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence
  * Zhang et al. (2020b) Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L (2020b) Ifcnn: A general image fusion framework based on convolutional neural network. Information Fusion 54:99–118
  * Zhao et al. (2023a) Zhao W, Xie S, Zhao F, He Y, Lu H (2023a) Metafusion: Infrared and visible image fusion via meta-feature embedding from object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13955–13965
  * Zhao et al. (2023b) Zhao Z, Bai H, Zhang J, Zhang Y, Xu S, Lin Z, Timofte R, Van Gool L (2023b) Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5906–5916