# Interaction of quantum systems with single pulses of quantized radiation

Victor Rueskov Christiansen<EMAIL_ADDRESS>Center for Complex Quantum
Systems, Department of Physics and Astronomy, Aarhus University, Ny Munkegade
120, DK-8000 Aarhus C, Denmark  Alexander Holm Kiilerich  Numerical Competence
Centre, Ørsted A/S,
Kraftværksvej 53, Skærbæk, DK-7000 Fredericia, Denmark  Klaus Mølmer
<EMAIL_ADDRESS>Aarhus Institute of Advanced Studies, Aarhus University,
Høegh-Guldbergs Gade 6B, DK-8000 Aarhus C, Denmark  Center for Complex Quantum
Systems, Department of Physics and Astronomy, Aarhus University, Ny Munkegade
120, DK-8000 Aarhus C, Denmark

(August 29, 2024)

###### Abstract

The interaction of a propagating pulse of quantum radiation with a localized
quantum system can be described by a cascaded master equation with a distinct
initially populated input and a finally populated output field mode [Phys.
Rev. Lett.123, 123604 (2019)]. By transformation to an appropriate interaction
picture, we identify the usual Jaynes-Cummings Hamiltonian between the
scatterer and a superposition of the initial and final mode, with a strength
given by the travelling pulse mode amplitude. The transformation also
identifies a coupling of the scatterer with an orthogonal combination of the
two modes. The transformed master equation offers important insights into the
system dynamics and it permits numerically efficient solutions.

##  I Introduction

A full quantum analysis of the interaction of a discrete quantum system with a
cavity field may often assume (near) resonance with only a single standing
wave field mode whose mode function is unaffected by the interaction. The
interaction then only implies the joint evolution of a single field oscillator
and the discrete quantum system, as described, e.g., in the Jaynes-Cummings
model.

Numerous proposals exist to employ interactions with simple quantum systems
with the purpose to manipulate and prepare quantum states, such as number
states, coherent states, squeezed states and Schrödinger cat states of a
single mode of radiation. The motivation is often that such states can be
propagated in space and thus be used for probing, communication, and transfer
of states and operations in quantum networks [1]. But, how does a localized
quantum system, such as a single two-level atom, interact with a travelling
pulse that initially occupies only a single mode of radiation? If the temporal
profile of this pulse at the location of the atom is $u(t)$, do we obtain a
time-dependent interaction of the same form as for an atom flying through a
cavity and exploring the position dependent field strength of the resonant
eigenmode? That is, is the system correctly or to a good approximation
described by the Hamiltonian $(\hbar=1)$,

$H_{JC}(t)=i\sqrt{\gamma}u(t)(\hat{a}^{\dagger}\sigma^{-}-\hat{a}\sigma^{+}),$
(1)

where $\sqrt{\gamma}u(t)$ specifies the time dependent coupling strength, and
the raising and lowering operators of the field and the two level system
account for the coherent exchange by absorption and emission of quanta of
excitation between the atom and the travelling pulse?

The answer to this question is complicated by the fact that in the absence of
a cavity, the propagating field explores a continuum of frequency modes and by
dispersive and absorptive effects, the interaction with a scatterer may change
the temporal shape of the field mode function in a manner that is entangled
with its quantum state of excitation. Multiple analyses have addressed
different aspects of this multi-photon, multi-mode problem [2, 3, 4, 5, 6, 7,
8, 9, 10, 11].

As a simpler problem, we may enquire what is the quantum state occupying a
definite pulse mode after the interaction. Applying the theory of cascaded
quantum system [12, 13], that problem was treated in Refs.[14, 15] by
incorporating the incident pulse of quantized radiation as the output from an
upstream virtual single mode cavity that gradually releases its quantum state
content in the form of a pulse. The quantum state of any specific output pulse
mode may, in a similar manner, be associated with the asymptotic final content
of downstream filter cavity. This experimentally inspired construction of
input and output wave packet modes leads to a simple density matrix formalism
with time dependent couplings of the scatterer to two discrete cavity modes.
The problem thus takes a quite different form than suggested by Eq.(1).

The cascaded master equation describes dynamics where the upstream cavity
(representing the incident pulse) leaks all its quanta, while the downstream
cavity gradually acquires the entire fraction of the output field that
populates the specified output pulse mode function. In this article we derive
the master equation in the interaction picture with respect to the undisturbed
propagation of the pulse, represented by a linear transfer of quanta between
the upstream and downstream cavities. In this interaction picture, the
coupling of the field to the scatterer takes a simple form and numerical
solutions are much less demanding.

The article is structured as follows: In section II the virtual cavitites and
cascaded system formalism is presented. In section III we introduce the
interaction picture and the corresponding master equation for the case of
identical input and output modes. In section IV we extend the interaction
picture to describe different input and output pulses, which includes for
example the effect of dispersion caused by an empty cavity. As a numerical
example we analyze the creation of squeezed states and Schrödinger cat states
by the scattering of pulses on a cavity with a Kerr non-linearity.

##  II Quantum interactions with a light pulse - a virtual cavity approach

Consider a local quantum system described by the Hamiltonian
$\hat{H}_{\textup{s}}(t)$ and Lindblad dissipation terms $\hat{L}_{i}$, with
$i=1,...n$. Its interaction with an incident quantized radiation field is
governed by
$V=i\sqrt{\gamma}[\hat{c}\hat{b}_{in}(t)^{\dagger}-\hat{c}^{\dagger}\hat{b}_{in}(t)]$,
where $\hat{c}^{(\dagger)}$ is the lowering (raising) operator of the system
accompanying absorption (emission) of a quantum of radiation. This interaction
involves the input field continuum operators $\hat{b}_{in}(t)$ which can be
viewed as the Fourier transform of the frequency eigenmode annihiliation
operators. If the incident radiation is restricted to a single pulse,
described by a square integrable mode function $u(t)$, the interaction can be
accounted for by an effective cascaded system master equation [14]. Here the
quantum pulse is described as if it leaks from an upstream virtual cavity with
a coherent out-coupling strength $g_{u}(t)$, where

$\displaystyle
g_{u}(t)=\frac{u^{*}(t)}{\sqrt{1-\int_{0}^{t}dt^{\prime}\,|u(t^{\prime})|^{2}}}.$
(2)

Likewise the component of outgoing radiation that eventually occupies an
arbitrary wave packet mode $v(t)$, can be picked up by a virtual downstream
filter cavity with a coherent in-coupling strength $g_{v}(t)$,

$\displaystyle
g_{v}(t)=-\frac{v^{*}(t)}{\sqrt{\int_{0}^{t}dt^{\prime}\,|v(t^{\prime})|^{2}}}.$
(3)

All other output modes are reflected by the $v$-cavity, and they are, in our
formalism described as Markovian loss. According to input-output theory [12],
the output field after reflection by the $v$-cavity is thus governed by the
annihilation operator

$\displaystyle\hat{b}_{out}(t)=\hat{b}_{in}(t)+g_{u}^{*}(t)\hat{a}_{u}+\sqrt{\gamma}\hat{c}+g_{v}^{*}(t)\hat{a}_{v}.$
(4)

which represents the interference between the amplitudes of the incident
vacuum field operator and fields emitted by the scatterer and the two
cavities. Detection of an outgoing photon is thus accompanied by the action of
a single quantum jump operator, equivalent to the appearance of a Lindblad
damping term,

$\displaystyle\hat{L}_{0}(t)=\sqrt{\gamma}\hat{c}+g_{u}^{*}(t)\hat{a}_{u}+g_{v}^{*}(t)\hat{a}_{v},$
(5)

in the master equation for the joint state of the quantum scatterer and the
two virtual cavities ($\hbar=1$),

$\displaystyle\frac{d\rho}{dt}=\frac{1}{i}[\hat{H}(t),\rho]+\sum_{i=0}^{n}D[\hat{L}_{i}]\rho.$
(6)

Here, the Hamiltonian is formed both by the system part
$\hat{H}_{\mathrm{s}}(t)$ and the interactions between the different
components,

$\displaystyle\begin{split}&\hat{H}(t)=\hat{H}_{\mathrm{s}}(t)+\frac{i}{2}\big{(}\sqrt{\gamma}g_{u}(t)\hat{a}_{u}^{\dagger}\hat{c}+\sqrt{\gamma}g_{v}^{*}(t)\hat{c}^{\dagger}\hat{a}_{v}+g_{u}(t)g_{v}^{*}(t)\hat{a}_{u}^{\dagger}\hat{a}_{v}-\mathrm{H.c.}\big{)}\end{split}$
(7)

and the master equation terms
$D[\hat{L}_{i}]\rho=-\frac{1}{2}(\hat{L}_{i}^{\dagger}\hat{L}_{i}\rho+\rho\hat{L}_{i}^{\dagger}\hat{L}_{i})+\hat{L}_{i}\rho\hat{L}_{i}^{\dagger}$
apply both for the outgoing field loss, represented by $\hat{L}_{0}$ in (5)
and for the damping terms $\\{\hat{L}_{i=1,..n}\\}$ acting on the quantum
scatterer. Note that both the Hamiltonian and the
$\hat{L}_{0}^{\dagger}\hat{L}_{0}$ product terms in the master equation give
rise to cross terms between the field and scatterer operators
$\hat{a}_{u(v)}^{(\dagger)}$ and $\hat{c}^{(\dagger)}$, and the Hamiltonian
and the dissipative terms conspire to cancel all contributions that cause
excitation transfer towards the upstream cavity. This is a key property of the
cascaded system master equation, built into its formal derivation [12, 13].

Reflection by a one-sided cavity is treated in a similar manner and it is also
possible to treat multi-output situations where a scatterer causes both
transmission and reflection [15]. If retardation effects can be neglected,
more complex networks can also be treated [16], while non-Markovian effects of
retardation and time delays engages the multi-mode character of the field
[17].

##  III Time dependent modes - an interaction picture

The introduction of separate upstream and downstream cavity modes in our
analysis enforces a two-mode treatment of the interaction between a single
light pulse and a quantum system. Since the bare propagation of the light
pulse without a scatterer amounts to a perfect release and recapture of the
field by the input and output cavity modes, this suggest addressing the
dynamics in an interaction picture where the transfer of the quantum state
between these modes is already taken care of. The analysis of the problem
gives exactly the same results, but the interaction picture should allow a
much more efficient numerical treatment.

By the above argument, we hence pass to the interaction picture with respect
to the cavity-cavity coupling component of (7),

$\displaystyle\hat{H}_{0}(t)=\frac{i}{2}[g_{u}(t)g_{v}^{\ast}(t)\hat{a}_{u}^{\dagger}\hat{a}_{v}-g_{v}(t)g_{u}^{\ast}(t)\hat{a}_{v}^{\dagger}\hat{a}_{u}].$
(8)

We thus define the unitary time evolution operator $\hat{U}_{0}(t)$ as the
solution to,

$\displaystyle i\frac{d}{dt}\hat{U}_{0}(t)=\hat{H}_{0}(t)\hat{U}_{0}(t)$ (9)

with $\hat{U}_{0}(0)=\hat{I}$, the identity operator. The Schrödinger picture
solution to the master equation can be written
$\rho(t)=\hat{U}_{0}(t)\rho_{I}(t)\hat{U}_{0}(t)^{\dagger}$, where
$\rho_{I}(t)$ solves the master equation in the interaction picture. In the
interaction picture, $\hat{H}_{0}(t)$ is absent from the Hamiltonian, while
the remaining terms and the Lindblad operators are transformed,
$\hat{O}_{I}(t)=\hat{U}_{0}(t)^{\dagger}\hat{O}\hat{U}_{0}(t)$.

We recognize $\hat{H}_{0}(t)$ as a time dependent beam splitter-type coupling,
and we make the ansatz

$\displaystyle\hat{U}_{0}(t)=\exp(\lambda(t)\hat{a}_{u}^{\dagger}\hat{a}_{v}-\lambda^{\ast}(t)\hat{a}_{v}^{\dagger}\hat{a}_{u}),$
(10)

which upon insertion in Eq.(9) yields
$\frac{d}{dt}\lambda(t)=\frac{1}{2}g_{u}(t)g_{v}^{\ast}(t)$. In the
interaction picture, the cavity mode annihilation operators thus read

$\displaystyle\begin{split}\hat{a}_{u,I}(t)&=\cos\lambda(t)\hat{a}_{u}(0)+\sin\lambda(t)\hat{a}_{v}(0)\\\
\hat{a}_{v,I}(t)&=\cos\lambda(t)\hat{a}_{v}(0)-\sin\lambda(t)\hat{a}_{u}(0),\end{split}$
(11)

where $\hat{a}_{u(v)}(0)$ refer to the incident quantum field and the vacuum
output field at the initial time, where the Schrödinger and the Interaction
picture coincide. Note that the bare system Hamiltonian
$\hat{H}_{\mathrm{s}}(t)$ and damping terms $\hat{L}_{i}^{\dagger}$,
$\hat{L}_{i}$ with $i>0$ are not affected by the transformation to the
interaction picture, which only concerns the pulse mode operators. We thus
obtain

$\displaystyle\hat{H}_{I}=\hat{H}_{S}+\frac{i\sqrt{\gamma}}{2}((g_{u}(t)\cos\lambda^{\ast}+g_{v}(t)\sin\lambda^{\ast})\hat{a}_{u}^{\dagger}\hat{c}+(g_{v}^{\ast}(t)\cos\lambda-
g_{u}^{\ast}(t)\sin\lambda)\hat{c}^{\dagger}\hat{a}_{v}-\text{H.c}),$ (12)

and

$\displaystyle\hat{L}_{0,I}=\sqrt{\gamma}\hat{c}+g_{u}^{\ast}(t)(\cos\lambda\hat{a}_{u}+\sin\lambda\hat{a}_{v})+g_{v}^{\ast}(t)(\cos\lambda\hat{a}_{v}-\sin\lambda\hat{a}_{u})$
(13)

In the following we shall omit the index $I$ with the understanding that all
explicitly time dependent operators refer to the interaction picture.

###  III.1 Identical incoming and outgoing modes

We examine first the case where the outgoing field mode is equal to the
incoming one, $v(t)=u(t)$, which suggests introducing

$\displaystyle\sin^{2}\theta(t)\equiv\int_{0}^{t}dt^{\prime}|u(t^{\prime})|^{2},$
(14)

with $|u(t)|^{2}=d\sin^{2}\theta/dt=2\sin\theta\cos\theta d\theta/dt$, since
this allows us to rewrite

$\displaystyle
g_{u}^{*}(t)g_{v}(t)=\frac{-u(t)^{2}}{\sin\theta(t)\cos\theta(t)}=-2\frac{d\theta}{dt}.$
(15)

We thus obtain the simple relation, $\lambda(t)=-\theta(t)$. When $u(t)$ is
real, we can rewrite the interaction picture Hamiltonian (12) as

$\displaystyle\hat{H}(t)=\hat{H}_{S}(t)+i\sqrt{\gamma}u(t)(\hat{a}_{u}^{\dagger}\hat{c}-\hat{c}^{\dagger}\hat{a}_{u})+\frac{i}{2}\sqrt{\gamma}u(t)(\cot\theta-\tan\theta)(\hat{a}_{v}^{\dagger}\hat{c}-\hat{c}^{\dagger}\hat{a}_{v}),$
(16)

and the Lindblad operator (13) of field loss as

$\displaystyle\hat{L}_{0}(t)=\sqrt{\gamma}\hat{c}-(\tan\theta+\cot\theta)u(t)\hat{a}_{v}.$
(17)

Remarkably, the first interaction term in Eq. (16) is exactly the Jaynes-
Cummings Hamiltonian (1) that one might have anticipated by simple arguments,
but the interaction with the travelling pulse is supplemented by the coupling
to a second mode of the field, and the dissipation of the system (17) includes
also this ancillary mode.

Figure 1: a) A pulse $u(t)$ is emitted from a virtual cavity with coupling
$g_{u}(t)$ and interacts with a localized quantum system. Another virtual
cavity with coupling $g_{v}(t)$ absorbs the quantum state contents of the
outgoing pulse, chosen here of the same form $u(t)$. b) In the interaction
picture, the scatterer interacts with two time dependent modes $a_{u}(t)$ and
$a_{v}(t)$ with interaction strengths $\sqrt{\gamma}u(t)$ and
$\frac{\sqrt{\gamma}}{2}u(t)(\cot\theta-\tan\theta)$. The operators
$\hat{a}_{u(v)}(t)$ act on time dependent superpositions of the incident and
outgoing modes in panel (a), and they are graphically represented here by the
left and right cavities. $\hat{a}_{u(v)}(t)$ coincide initially with the
Schrödinger picture mode operators $\hat{a}_{u(v)}(0)$ but with time they
exchange character and become $\hat{a}_{v(u)}(0)$.

A visual representation of the transformation to the interaction picture is
presented in Fig. 1, where panel (a) shows the release and recapture of the
pulse $u(t)$ by the virtual cavities with time dependent coupling strengths
$\sqrt{\gamma}g_{u}(t)$ and $\sqrt{\gamma}g_{v}(t)$ to the scatterer in the
Schrödinger picture. Panel (b) shows the coupling to the time evolved modes in
the interaction picture, where the left most cavity represents the freely
propagating pulse mode, cf., the coupling strength $\sqrt{\gamma}u(t)$, and
the right most cavity represents an orthogonal superposition of the two modes
shown in panel (a).

In absence of the coupling to the central system (setting $\gamma=0$) the
Hamiltonian vanishes in the interaction picture, and nothing happens to the
quantum content of the time dependent left and right modes in the interaction
picture of Fig. 1(b). When $\gamma\neq 0$, during the finite interaction time,
there may be only a limited exchange of quanta between the scatterer and the
time dependent modes which is in stark contrast with the complete emptying and
partial filling of the $u$\- and $v$-cavity modes in the Schrödinger picture
represented by Fig. 1(a).

###  III.2 In what direction does the radiation propagate in the interaction
picture ?

In our cascaded open system approach [14, 15], it is assumed, that the light
only travels – and the central system only emits – in the direction from the
initially populated $u$-cavity towards the output mode $v$-cavity.

Figure 2: Decay of an excited two level system into a travelling pulse $u(t)$,
treated here as an input vacuum pulse, and output single photon pulse. The
interaction is described by a time dependent Jaynes-Cummings interaction and
the coupling and correlated decay with an ancillary mode.

In the Schrödinger picture, terms in the Hamiltonian and the Lindblad damping
terms thus exactly cancel upstream propagation of excitations from the
scatterer and the $v$-cavity. The same formal cancellation, however, is less
obvious in the interaction picture where an initial excitation of the
$v$-cavity seems to play a role in the dynamics due to the
$\hat{c}^{\dagger}\hat{a}_{v}$ term in the Hamiltonian (16). A numerical
calculation, however, shows that the $v$-cavity is instantly emptied, and to
derive this result, we consider the mean amplitude $\beta$ in the $v$-cavity
mode, which decays by the rate

$\displaystyle\frac{d\beta}{dt}=-\frac{|g_{v}|^{2}}{2}\beta.$ (18)

Solving this equation by quadrature, we obtain

$\displaystyle\int_{\beta_{1}}^{\beta_{2}}\frac{d\beta}{\beta}=-\frac{1}{2}\int_{0}^{t}dt^{\prime}|g_{v}(t^{\prime})|^{2}.$
(19)

Now, we note that $|g_{v}(t)|^{2}$ equals $|v(t)|^{2}$ divided by
$F(t)=\int_{0}^{t}dt^{\prime}|v(t^{\prime})|^{2}$, and we define
$f(t)=|v(t)|^{2}=dF/dt$ to obtain

$\displaystyle\int_{\beta_{1}}^{\beta_{2}}\frac{d\beta}{\beta}=\ln{\frac{\beta_{2}}{\beta_{1}}}=-\frac{1}{2}\int_{0}^{t}dt^{\prime}\frac{f(t^{\prime})}{F(t^{\prime})}.$
(20)

The last term is readily evaluated and yields

$\displaystyle\ln{\frac{\beta(t)}{\beta(0)}}=-\frac{1}{2}\ln{\left(\frac{F(t)}{F(0)}\right)}\Rightarrow\beta(t)=\beta(0)\sqrt{\frac{F(0)}{F(t)}}.$
(21)

Since $F(t)$ vanishes for $t=0$ and is finite for any $t>0$, any initial
excitation in the $v$-cavity, indeed, decays instantly and there is no
upstream propagation.

Figure 3: Simulation of Rabi oscillation dynamics due to excitation of a two-
level system by an $n=20$ Fock state Gaussian pulse. Panel (a) shows how the
interaction picture $u$ oscillator is subject to the exchange of only few
quanta of excitation with the scatterer and the $v$-oscillator. Panel (b)
shows how the two level atom undergoes partial Rabi oscillations, out of phase
with the mean excitation of the $u$-mode, confirming the exhange of quanta
with the travelling pulse. The ancillary $v$-mode is temporarily excited while
the net decrease of the total number of excitations in the final state is due
to incoherent loss to other modes and a possible coherent modification of the
output pulse due to the interaction.

###  III.3 Rabi Oscillations with a quantum pulse

Atoms excited by coherent monochromatic light show sinusoidal Rabi
oscillations of their excited state population,
$\rho_{22}(t)=\sin^{2}\left(\Omega t/2\right)$. In cavity QED, similar
oscillations occur for an atom interacting with a quantized field [18], and we
can now also investigate how an atom behaves when exposed to a resonant pulse
$u(t)$ with quantum light.

We want to address the excitation of both the atom and of the travelling
pulse, and we consider the output field mode $v(t)=u(t)$, described by the
Hamiltonian (16) and the Lindblad operator (17). We use a normalized Gaussian
pulse

$\displaystyle
u(t)=\frac{1}{\sqrt{\tau}\pi^{1/4}}\text{exp}\left(-\frac{(t-t_{p})^{2}}{2\tau^{2}}\right),$
(22)

with $t_{p}=4\gamma^{-1}$ and $\tau=1\gamma^{-1}$ in the numerical
calculations. With this choice Eq.(14) yields

$\displaystyle\theta(t)=\sin^{-1}\left(\sqrt{\frac{1}{2}\left(\text{erf}\left(\frac{t-t_{p}}{\tau}\right)+\text{erf}\left(\frac{t_{p}}{\tau}\right)\right)}\right),$
(23)

where erf denotes the error function.

Fig. 3 shows the time evolution, starting with the ground state atoms and an
incident pulse prepared in a Fock state with $n=20$ photons. It is clear from
the behavior of $\langle\hat{c}^{\dagger}\hat{c}\rangle$ in the lower panel
that the two-level system undergoes three Rabi oscillations before the
interaction is over and the system relaxes to the ground state. In the
Schrödinger picture calculation, the $u$-cavity mode is completely emptied of
its initial excitation which is partially retrieved by the $v$-cavity mode,
and the quantum state of the field hence explores the tensor product of two
21-dimensional Hilbert spaces, In the interaction picture, the time evolving
oscillator modes explore only a few different Fock states. The upper panel in
Fig.(3) thus shows how the time dependent $u$-mode loses just 1-2 photons
while the $v$-mode acquires less than unit excitation. Truncation of the
Hilbert space to the relevant number state components thus easily yields a
20-fold reduction in dimension, and a corresponding 400-fold reduction in the
number of density matrix elements. A factor that would be even larger for
larger incident photon numbers and results in much shorter computing times.

##  IV Three-mode interaction picture

In the previous section, we transformed to an interaction picture with respect
to the coupling term between virtual $u$ and $v$, input and output cavity
modes. If the scatterer is itself an optical cavity or a system contained
inside a cavity, it is possible to apply an interaction picture with respect
to the coupling terms among all these cavities. For a linear coupling of three
cavities modes described by the Hamiltonian $\hat{H}_{0}$

$\displaystyle\hat{H}_{0}(t)=\frac{i}{2}(\sqrt{\gamma}g_{u}(t)\hat{a}_{u}^{\dagger}\hat{c}+\sqrt{\gamma}g_{v}^{\ast}(t)\hat{c}^{\dagger}\hat{a}_{v}+g_{u}(t)g_{v}^{\ast}(t)\hat{a}_{u}^{\dagger}\hat{a}_{v}-\text{H.c.}),$
(24)

we define $\hat{U}_{0}(t)$ as the solution to Eq. (9), and assume the ansatz

$\displaystyle\hat{U}_{0}(t)=\text{exp}(\lambda_{1}(t)\hat{a}_{u}^{\dagger}\hat{c}+\lambda_{2}(t)\hat{c}^{\dagger}\hat{a}_{v}+\lambda_{3}(t)\hat{a}_{u}^{\dagger}\hat{a}_{v}-\text{H.c.}).$
(25)

From the Hamiltonian (24) and the Ansatz for the unitary evolution (25),
follows the coupled differential equations,

$\displaystyle\dot{\lambda}_{1}(t)=\frac{1}{2}\sqrt{\gamma}g_{u}(t),\quad\dot{\lambda}_{2}(t)=\frac{1}{2}\sqrt{\gamma}g_{v}^{\ast}(t),\quad\dot{\lambda}_{3}(t)=\frac{1}{2}g_{u}(t)g_{v}^{\ast}(t).$
(26)

We can expand the set of field operators in the interaction picture as a
vector

$\displaystyle\phi(t)=\begin{pmatrix}\hat{a}_{u}(t)\\\ \hat{c}(t)\\\
\hat{a}_{v}(t)\end{pmatrix}=M(t)\phi(0),$ (27)

where $\phi(0)$ is the vector of operators in the Schrödinger picture.

The Ansatz for the interaction picture operators yields the linear system of
equations

$\displaystyle\frac{d}{dt}M(t)=F(t)M(t),$ (28)

where the coefficient matrix $F(t)$ contains the derivatives of the
$\lambda$-functions according to (26)

$\displaystyle
F(t)=\frac{1}{2}\begin{pmatrix}0&\sqrt{\gamma}g_{u}(t)&g_{u}(t)g_{v}^{\ast}(t)\\\
-\sqrt{\gamma}g_{u}^{\ast}(t)&0&\sqrt{\gamma}g_{v}^{\ast}(t)\\\
-g_{u}^{\ast}(t)g_{v}(t)&-\sqrt{\gamma}g_{v}(t)&0\end{pmatrix}.$ (29)

This linear system of equation is readily solved and yields the transformation
of any further components of the Hamiltonian to the interaction picture.

###  IV.1 Three-mode interaction picture with an empty cavity

We assume three cavities with the ladder operators $\hat{a}_{u}$, $\hat{c}$
and $\hat{a}_{v}$ and a Gaussian $u$-pulse as specified in section III.3. In
the absence of any further interactions, the transformation to the interaction
picture handles all the dynamics, and no quanta ever leave the interaction
picture $\hat{a}_{u}$-mode while the interaction picture $\hat{c}$ and
$\hat{a}_{v}$ are “dark” modes that never become populated during the
interaction.

Note that the output mode after scattering on a single mode cavity with
linewidth $\gamma$ and no internal losses is given in frequency space by
$v(\omega)=r(\omega)u(\omega)$ [14] where

$\displaystyle
r(\omega)=\frac{i(\omega-\omega_{c})+\frac{\gamma}{2}}{i(\omega-\omega_{c})-\frac{\gamma}{2}}.$
(30)

At late times, the interaction picture operator $\hat{a}_{u}$ accounts for all
the incident photons which now populate the pulse mode transformed according
to (30).

###  IV.2 Three-mode interaction picture with a Kerr non-linear cavity:
squeezing of a light pulse

As a non-trivial example of the application of the three-mode interaction
picture, we introduce a non-linear Kerr effect in the $c$-cavity,

$\displaystyle H_{s}(t)=K(\hat{c}^{\dagger}(t)\hat{c}(t))^{2},$ (31)

where $K$ is a constant. The non-linear Kerr interaction
$(\hat{c}^{\dagger}(t)\hat{c}(t))^{2}$ [19] acts on a coherent state by phase
shifting each Fock state by an amount proportional to $n$. This effectively
stretches the complex phase space distribution of the state and transforms the
coherent state into a squeezed state. This picture readily applies to an
intra-cavity field, but a long pulse incident on a cavity may at no time have
all its photons inside the cavity and may thus not be subject to the full non-
linear interaction, while a short pulse is spectrally broad and may reflect
without even entering the cavity.

Our theory takes the spatial propagation properly into account, and our
interaction picture restricts the evolution of the quantum states of the field
to the number states initially occupied. We have found that using
$K=0.02\gamma$, this effect transforms a coherent state with $\alpha=4$ and
the pulse parameters defined in section III.3 into a squeezed state at the end
of the interaction. States with larger coherent state amplitudes become more
squeezed while larger values of the Kerr-interaction strength $K$ may distort
the mode shape and cause deterioration of the single mode character and hence
loss of squeezing.

We quantify the degree of squeezing by minimizing the variance of field
quadratures rotated by the phase angle $\phi$,

$\displaystyle\text{Var}(x_{\phi})=\left\langle\left[\Delta(\cos{\phi}\hat{x}-\sin{\phi}\hat{p})\right]^{2}\right\rangle.$
(32)

For the case of an interaction with $K=0.02\gamma$ and $\alpha=4$, the minimum
uncertainty arises for the angle $\phi=0.52$ rad. where the variance is
$\langle[\Delta(\cos{\phi}\hat{x}-\sin{\phi}\hat{p})]^{2}\rangle\approx 0.245$
which is squeezed compared to the coherent state value of 0.5. Fig.4 shows the
Wigner function indeed resembles that of a squeezed state along an axis
rotated clockwise by $\phi=0.52\;\mathrm{rad}\approx 30\degree$. In Fig. 5,
the variance is shown as a function of $\phi$ for different values of the Kerr
interaction strength, and we observe that the stronger Kerr effect leads to a
smaller degree of (single mode) squeezing.

Figure 4: The Wigner function of an incident coherent state with $\alpha=4$
($\langle x\rangle=4\sqrt{2}$) (left panel) and the squeezed output state
after passage of the pulse through a resonator with a Kerr-interaction with
strength $K=0.02\gamma$ (right panel). Figure 5: The variance of rotated field
quadrature variables of a light pulse after passage of a non-linear resonator
with different values of the Kerr non-linearity $K$. Values below the dashed
line at the value 0.5 meet the the “squeezing condition”.

###  IV.3 Three-mode interaction picture with a Kerr non-linear cavity:
turning a coherent pulse into a Schrödinger cat

The value of $n^{2}$ can be written as $4m$ and $4m+1$ for even and odd $n$
respectively, where $m$ is an integer. For a sufficiently strong Kerr-
interaction, the accumulated different phase factors attain the values $i$ on
odd and $1$ on even Fock states and hence transfer a coherent state in a
single mode cavity into a Schrödinger cat state [20].

In our set-up, only the component of the pulse which is inside the non-linear
$c$-mode cavity experiences the Kerr-interaction, which must be strong enough
to yield the discrete phase differences to form the cat state, yet weak enough
that it does not ruin the single mode character of the pulse. This poses a
useful application of the interaction picture calculation, which focuses on
the field content of the travelling pulse and takes the linear dispersion of
the pulse shape by the passage of the middle cavity into account. In the
absence of the Kerr interaction, the quantum state of the pulse is fully
preserved and unchanged. Assuming the time-dependent operators in the
interaction picture with respect to (24) and the expansion (27), for a weak
Kerr interaction we may estimate its effect on the quantum state, exploiting
the expression for the $\hat{c}(t)$-operator

$\displaystyle\hat{c}(t)=M_{21}(t)\hat{a}_{u}(0)+M_{22}(t)\hat{c}(0)+M_{23}(t)\hat{a}_{v}(0).$
(33)

If we approximate $\hat{c}(t)$ by including only the contribution from the
initially populated $\hat{a}_{u}$-mode, we have
$(\hat{c}^{\dagger}(t)\hat{c}(t))^{2}\approx|M_{21}(t)|^{4}(\hat{a}_{u}^{\dagger}\hat{a}_{u})^{2}=|M_{21}(t)|^{4}\hat{n}^{2}$,
where the operator $\hat{n}^{2}$ is the square of the almost unchanged photon
number in the pulse. The solution to Schrödinger’s equation (in the
interaction picture) for an initial coherent state is then

$\displaystyle\ket{\psi(t)}=\text{exp}\left(-iK\hat{n}^{2}\int_{0}^{t}dt^{\prime}|M_{21}(t^{\prime})|^{4}\right)\ket{\alpha}.$
(34)

To create the Yurke-Stoler state, the phase difference between even and odd
$n$ should be $\pi/2$, and hence $K$ must satisfy

$\displaystyle
K=\frac{\pi}{2}\left(\int_{0}^{T}dt^{\prime}|M_{21}(t^{\prime})|^{4}\right)^{-1},$
(35)

where $T$ is the duration of the pulse. Assuming an incident Gaussian wave
packet with the same parameters as in section III.3, the numerical evaluation
of the integral over $M_{21}(t)$ yields a required value of $K\simeq 1.3$ to
form the Schrödinger cat state. When we solve the three-mode problem
numerically in the interaction picture with a sufficiently strong interaction,
however, we find a significant loss of population of the $\hat{a}_{u}$ pulse
modes, see Fig.6. Fig. 7 shows that the Wigner function of the quantum state
of the travelling pulse-mode at different times during the interaction. It is
evident that the loss of amplitude also prevents the formation of the cat
state.

Figure 6: The upper panel shows the input $u(t)$ and output $v(t)$ modes for
the scattering on an empty cavity. The lower panel shows the occupation of the
output mode $v(t)$, represented asymptotically by $\hat{a}_{u}$ in the
interaction picture, for different values of the non-linear interaction $K$.
We assume an incident coherent state with
$\langle\hat{n}\rangle=|\alpha|^{2}=4$, and no photons are lost in the
process, but an increasing number of photons explore the continuum of modes
orthogonal to $v(t)$. Figure 7: The Wigner function of the travelling pulse-
mode at different times during the Kerr interaction, given by eq. (35) for an
initial coherent state with $\alpha=2$. It is clear that the interaction does
not lead to a Schrödinger cat state.

It seems that the large value needed for $K$ is not compatible with the
preservation of the single mode character of the quantum field. Instead of
only looking at a single interaction, we therefore propose to let the light
pulse pass the non-linear $c$-cavity several times (or pass through a sequence
of several such cavities), and thus accumulate the non-linear phase shift from
repeated weak Kerr-interactions with a small value of $K$. To ensure that the
input pulse has the same initial Gaussian shape at each interaction, one may
need to reshape the pulse, e.g., by sum frequency generation [21]. The weak
Kerr interaction at each passage, transforms the quantum state of the field
only slightly, while maintaining its single mode character, and after
sufficiently many passages, the light pulse attains the Schrödinger cat
quantum state.

To create the cat by $N$ transmission steps, Eq. (35) can now be relaxed to

$\displaystyle N\cdot
K\int_{0}^{T}dt^{\prime}|M_{21}(t^{\prime})|^{4}=\frac{\pi}{2}.$ (36)

Using an interaction strength $K=0.01\gamma$, a Gaussian wave packet with the
same parameters as in section III.3 and the numerical evaluation of the
integral $\int_{0}^{T}dt^{\prime}|M_{21}(t^{\prime})|\approx
1.180\gamma^{-1}$, we estimate the number of interactions required to create a
cat state from an input coherent state as

$\displaystyle N=\frac{\pi}{2\cdot 0.01\gamma\cdot 1.180\gamma^{-1}}\approx
133.$ (37)

This is in good agreement with the full numerical calculation, presented in
Fig. 8, which shows the development of different non–classical states in the
process towards the final Yurke-Stoler cat state.

Figure 8: The Wigner function of the quantum state of the transmitted pulse
after different numbers of cavity transmissions. The simulations are carried
out for $K=0.01\gamma$ and a coherent state with $\alpha=2$, and they assume
that the output pulse of each cavity transmission is filtered and reshaped
before transmission through the next cavity.

##  V Conclusion and outlook

An effective cascaded open quantum system approach can be used to describe the
initial preparation, the interaction and the final analysis of how a pulse of
quantum radiation interacts with, e.g., an atom. The propagation of quantum
pulses in free space and among cavities can be solved exactly in the
Heisenberg picture by a linear transformation of the mode operators. This
constitutes a good starting point for an interaction picture treatment of the
cascaded master equation theory. The interaction between the propagating light
and localized scatterers thus takes the conventional Jaynes-Cummings form of a
time-dependent single mode interaction, while it also provides the interaction
with an auxiliary oscillator. These terms together ensure the exact analysis
assuming only the standard Born-Markov treatment of couplings to the continuum
of free space (or guided) modes.

Our theory is simple to implement directly with standard master equation
solvers, and since the main part of the dynamics is already taken care of by
the transformation to the interaction picture, it considerably simplifies the
numerical calculations. We note that our analysis may equally apply to other
bosonic fields (light, microwaves, matter waves, sound and spin waves). For a
wide range of wave phenomena, (linear) dispersion plays an important role and
may be treated by a suitable interaction picture along the same lines as our
treatment of the transmission through a single linear cavity. Our method thus
permits dealing separately with the effects of propagation and of quantum
interactions.

##  VI Acknowledgements

The authors acknowledge discussions with Fan Yang and Mads Middelhede Lund and
support from the Euro- pean Union FETFLAG program, Grant No. 820391 (SQUARE),
the Danish National Research Foundation through the Center of Excellence for
Complex Quantum Systems (Grant agreement No. DNRF156), and the European
Union’s Horizon 2020 Research and Innovation Programme under the Marie
Sklodowska-Curie program (754513).

## References

  * Kimble [2008] H. J. Kimble, “The quantum internet,” Nature 453, 1023 (2008).
  * Gheri  _et al._ [1998] K. M. Gheri, K. Ellinger, T. Pellizzari, and P. Zoller, “Photon-wavepackets as flying quantum bits,” Fortschritte der Physik: Progress of Physics 46, 401–415 (1998).
  * Shen and Shen [2015] Y. Shen and J.-T. Shen, “Photonic-Fock-state scattering in a waveguide-QED system and their correlation functions,” Phys. Rev. A 92, 033803 (2015).
  * Baragiola  _et al._ [2012] B. Q. Baragiola, R. L. Cook, A. M. Brańczyk, and J. Combes, “$n$-photon wave packets interacting with an arbitrary quantum system,” Phys. Rev. A 86, 013811 (2012).
  * Gough  _et al._ [2012] J. E. Gough, M. R. James, H. I. Nurdin, and J. Combes, “Quantum filtering for systems driven by fields in single-photon states or superposition of coherent states,” Phys. Rev. A 86, 043819 (2012).
  * Baragiola and Combes [2017] B. Q. Baragiola and J. Combes, “Quantum trajectories for propagating fock states,” Phys. Rev. A 96, 023819 (2017).
  * Paris-Mandoki  _et al._ [2017] A. Paris-Mandoki, C. Braun, J. Kumlin, C. Tresp, I. Mirgorodskiy, F. Christaller, H. P. Büchler, and S. Hofferberth, “Free-space quantum electrodynamics with a single Rydberg superatom,” Phys. Rev. X 7, 041010 (2017).
  * ang Guofeng Zhang and Wu [2021] Wen-Long Li ang Guofeng Zhang and Re-Bing Wu, “On the control of flying qubits,” (2021), arXiv:2111.00143 [quant-ph] .
  * Shen and Fan [2005] J. T. Shen and Shanhui Fan, “Coherent photon transport from spontaneous emission in one-dimensional waveguides,” Opt. Lett. 30, 2001–2003 (2005).
  * Fischer  _et al._ [2018] Kevin A. Fischer, Rahul Trivedi, Vinay Ramasesh, Irfan Siddiqi, and Jelena Vučković, “Scattering into one-dimensional waveguides from a coherently-driven quantum-optical system,” Quantum 2, 69 (2018).
  * Shi  _et al._ [2015] Tao Shi, Darrick E. Chang, and J. Ignacio Cirac, “Multiphoton-scattering theory and generalized master equations,” Physical Review A 92 (2015), 10.1103/physreva.92.053834.
  * Gardiner and Collett [1985] C. W. Gardiner and M. J. Collett, “Input and output in damped quantum systems: Quantum stochastic differential equations and the master equation,” Phys. Rev. A 31, 3761–3774 (1985).
  * Carmichael [1993] H. J. Carmichael, “Quantum trajectory theory for cascaded open systems,” Phys. Rev. Lett. 70, 2273–2276 (1993).
  * Kiilerich and Mølmer [2019] Alexander Holm Kiilerich and Klaus Mølmer, “Input-output theory with quantum pulses,” Phys. Rev. Lett. 123, 123604 (2019).
  * Kiilerich and Mølmer [2020] Alexander Holm Kiilerich and Klaus Mølmer, “Quantum interactions with pulses of radiation,” Phys. Rev. A 102, 023717 (2020).
  * Lodahl  _et al._ [2017] P. Lodahl, S. Mahmoodian, S. Stobbe, A. Rauschenbeutel, P. Schneeweiss, J. Volz, H. Pichler, and P. Zoller, “Chiral quantum optics,” Nature 541, 473 (2017).
  * Pichler and Zoller [2016] Hannes Pichler and Peter Zoller, “Photonic circuits with time delays and quantum feedback,” Phys. Rev. Lett. 116, 093601 (2016).
  * Agarwal [1985] G. S. Agarwal, “Vacuum-field rabi oscillations of atoms in a cavity,” J. Opt. Soc. Am. B 2, 480–485 (1985).
  * Gerry and P. [2005] C. Gerry and Knight P., “Introductory Quantum Optics,” (Cambridge University Press, New York, 2005) 1st ed.
  * Yurke and Stoler [1986] B. Yurke and D. Stoler, “Generating quantum mechanical superpositions of macroscopically distinguishable states via amplitude dispersion,” Phys. Rev. Lett. 57, 13–16 (1986).
  * Eckstein  _et al._ [2011] Andreas Eckstein, Benjamin Brecht, and Christine Silberhorn, “A quantum pulse gate based on spectrally engineered sum frequency generation,” Opt. Express 19, 13770–13778 (2011).


captionUnsupported document class

# Collaborative Federated Learning For Healthcare: Multi-Modal COVID-19
Diagnosis at the Edge

Adnan Qayyum1 Email<EMAIL_ADDRESS>Kashif Ahmad2 Muhammad Ahtazaz
Ahsan1 Ala Al-Fuqaha2 and Junaid Qadir1
1 Information Technology University (ITU)  Punjab  Lahore  Pakistan
2 Information and Computing Technologies (ICT) Division  College of Science
and Engineering (CSE)  Hamad Bin Khalifa University  Doha  Qatar

###### Abstract

Despite significant improvements over the last few years, cloud-based
healthcare applications continue to suffer from poor adoption due to their
limitations in meeting stringent security, privacy, and quality of service
requirements (such as low latency). The edge computing trend, along with
techniques for distributed machine learning such as federated learning, have
gained popularity as a viable solution in such settings. In this paper, we
leverage the capabilities of edge computing in medicine by analyzing and
evaluating the potential of intelligent processing of clinical visual data at
the edge allowing the remote healthcare centers, lacking advanced diagnostic
facilities, to benefit from the multi-modal data securely. To this aim, we
utilize the emerging concept of clustered federated learning (CFL) for an
automatic diagnosis of COVID-19. Such an automated system can help reduce the
burden on healthcare systems across the world that has been under a lot of
stress since the COVID-19 pandemic emerged in late 2019. We evaluate the
performance of the proposed framework under different experimental setups on
two benchmark datasets. Promising results are obtained on both datasets
resulting in comparable results against the central baseline where the
specialized models (i.e., each on a specific type of COVID-19 imagery) are
trained with central data, and improvements of 16% and 11% in overall
F1-Scores have been achieved over the multi-modal model trained in the
conventional Federated Learning setup on X-ray and Ultrasound datasets,
respectively. We also discuss in detail the associated challenges,
technologies, tools, and techniques available for deploying ML at the edge in
such privacy and delay-sensitive applications.

##  I Introduction

The main motivation for using data storage and computing at the edge stems
from the desire to make high quality computing resources available closer to
the users and to reduce the need for end devices to exchange private data with
centralized servers [satyanarayanan2017emergence]. There is an ongoing trend
to deploy machine learning (ML) algorithms at the edge enabling consumers and
corporations to enjoy and explore new opportunities in different application
domains, such as automotive, security, surveillance, and other smart city
services like healthcare [2]. This desire is particularly strong in the
healthcare industry where the stakes are high and various stakeholders
(including consumers, governments, and service providers) have stressed the
need for foolproof safeguards for ensuring data security, user privacy, and
ethical data use [1]. This motivates the use of edge computing in healthcare
settings to meet the high expected standards for patient privacy and security
as well as stringent requirements for quality of service (high reliability and
low latency).

Figure 1: An illustration of AI/ML at edge in an IoT empowered healthcare
environment.

Healthcare is a major application domain that can benefit from edge computing.
Generally, healthcare centers in remote areas lack advanced medical equipment
and other healthcare facilities resulting in poorer access to health services
by the people living there. Thanks to the recent advancement in telemedicine,
the provision of health services remotely, using audiovisual technology, is a
reality now. Large volumes of medical data including ultrasound and X-ray
images could be transmitted to major hospitals with advanced diagnosis
facilities for diagnostic and training purposes. However, there are several
challenges associated with a typical cloud-based infrastructure, such as low
bandwidth, high latency, transmission cost, and increasing concerns about data
security and privacy. Edge computing, which aims to store and process data
locally or closer to edge devices, on the other hand, results in low latency
and increased privacy and data security.

A typical IoT environment for smart healthcare is illustrated in Figure 1 in
which data collected by different sensors is processed at the edge for
different applications using ML techniques. Once the ML predicts an event, the
edge devices triggers an action or request service in the cloud. ML algorithms
can also be executed concurrently in the cloud as well as at the edge as shown
in the figure. However, with local data storage and processing resources
(through cloud computing), such applications enjoy a significant improvement
in terms of processing time by avoiding networking congestion. More
importantly, real-time processing at the edge improves the performance of
delay-sensitive applications, such as healthcare, surveillance and automotive,
by avoiding any potential latency or delay occurred during data transmission
between end devices and the cloud. In addition, ML at the edge results in
increased security and privacy as sending data back and forth from the cloud
may lead to security threats.

In this paper, we leverage the capabilities of edge computing in medicine by
analyzing and evaluating the potential of intelligent processing of clinical
visual data related to COVID-19 at the edge allowing the remote healthcare
centers to benefit from the multi-modal collaborative learning paradigm
without sharing any information about the modality of the local data and the
data itself. A number of recent research efforts have focused on diagnosing
COVID-19 using AI and data science methods [3]; relatively little work has
however focused on using edge AI for COVID-19 diagnosis.

To this aim, we utilize an emerging concept of clustered federated learning
(CFL) and propose a CFL-based collaborative learning framework for an
automatic multi-modal diagnosis of COVID-19. Our approach is well suited to
the task of COVID-19 diagnosis as visual data (i.e., CT scans, X-rays, and
ultrasound) is collected at different centers and could be used to build a
joint/shared ML model in a cloud-edge infrastructure able to diagnose COVID-19
in both X-ray and Ultrasound images. The proposed framework is evaluated on
two benchmark datasets under different experimental setups and we have
achieved encouraging results using CFL that are comparable with the baseline
results (when the model is trained with central data). We also discuss in
detail the potential applications, associated challenges, technologies, tools,
and techniques available for deploying ML at the edge in such privacy and
delay-sensitive applications. We note that we use the term multi-modal model
to represent a single model capable of diagnosing COVID-19 in both X-ray and
Ultrasound imagery when provided separately.

The main contributions of the paper are as follows:

  1. 1.

To highlight the potential of intelligent processing of clinical data at the
edge, we propose a collaborative learning framework for COVID-19 diagnosis by
leveraging a CFL approach enabling remote healthcare centers to benefit from
each other’s data without sharing the data itself and associated information.

  2. 2.

We also demonstrate how the performance of conventional FL is affected by the
divergence of the distribution of data from different sources (i.e., X-ray and
Ultrasound imagery), and how CFL can help to mitigate the adverse impact.

  3. 3.

We also highlight the potential challenges and enabling factors that enable
the deployment of ML/DL models to the edge.

  4. 4.

Finally, we elaborate on the open research issues related to deploying ML at
the edge for healthcare applications that require further investigation.

Organization of the paper: The rest of the paper is organized as follows.
Section II-A provides a broad discussion of the related work on automated
COVID-19 diagnosis as well as the different challenges encountered in
deploying ML on the edge along with a discussion on enabling technologies. The
case study for collaborative learning for multi-modal diagnosis of COVID-19 is
presented in Section III and results are presented in Section IV. Various open
research issues that require further investigation are presented in Section V.
Finally, Section VI provides some concluding remarks.

##  II Background

In this section, we provide background on the related work on automated
COVID-19 diagnosis as well as a discussion on the potential challenges that
hinder deployment ML at the edge along with the different enabling
technologies.

###  II-A Existing Automated COVID-19 Diagnosis Work

COVID-19 has been a strong focus of the research community in 2020, especially
after it was declared in March by the World Health Organization (WHO) to be a
pandemic, with diverse efforts focusing on diagnosis [4], treatment [5], and
the development of the potential vaccine [6]. Data science
methods—particularly, ML and data visualization techniques—are playing a major
role in the international response against the COVID-19 pandemic with some key
applications being risk assessment, contact tracking, fake news detection,
sentiment analysis, and screening and diagnosis [3]. The focus of this paper
is on automated screening and diagnosis; we shall discuss next some of the
prominent related techniques relying on different types of information (e.g.,
audio and visual data) that have been proposed.

A number of efforts have focused on automated image analysis in a bid to speed
up the COVID-19 diagnosis process [7]. To this aim, three different medical
imaging modalities, namely computerized tomography (CT), Ultrasound scans, and
X-radiation (X-ray), have been mostly exploited. To facilitate research on
image-based solutions for COVID-19 diagnosis, several datasets have been
collected and made publicly available [7, 9]. For instance, Maghdid et al. [9]
collected a comprehensive dataset containing a total of 170 X-rays and 361 CT
scan images from different sources. Cohen et al. [10] also provide a
collection of X-rays and CT scans of confirmed COVID-19 patients. A collection
of COVID-19 patients’ CT scans has also been made publicly available for
research purposes in [11, 12]. Born et al. [13], on the other hand, provide a
lung ultrasound (POCUS) dataset containing a total of 1103 images including
654 COVID-19, 277 bacterial pneumonia, and 277 healthy controls samples.

A vast majority of the image-based solutions for COVID-19 diagnosis relies on
CT scan images. For instance, Wan et al. [14] proposed a deep learning model
for extracting COVID-19’s specific features/textures in CT scans of confirmed
cases to extract useful clinical insight before the pathogenic tests. An
evaluation of a reasonable amount of confirmed cases showed encouraging
results with an average test accuracy of 73.1%. Butt et al. [15] proposed a
two-phase solution for COVID-19 diagnosis in CT scans. Initially, a pre-
trained 3D Convolutional Neural Network (CNN) is employed to extract potential
infectious regions in CT scans followed by a CNN-based classification
framework to classify the candidate regions into COVID-19, influenza, and non-
infectious regions. Li et al. [16] also proposed a 3D CNN-based framework to
extract both local and global deep features for diagnosis COVID-19 in CT
scans. One of the key challenges to CNN-based solution is the unavailability
of a large-scale CT scans datasets. In order to deal with the challenge,
Afshar et al. [17] proposed a Capsule Networks based deep learning framework,
namely COVID-CAPS, for COVID-19 diagnosis in X-ray images. Moreover, to
further enhance the capabilities of the proposed model, the authors used an
external dataset composed of 94, 323 frontal view chest X-ray images for pre-
training and transfer learning purposes.

There are also methods relying on X-ray images for COVID-19 diagnosis. For
instance, in [18] a pre-trained deep model is fine-tuned on X-ray images for
COVID-19 diagnosis. Similarly, Sethy et al. [19] trained a Support Vector
Machine (SVM) classifier on features extracted via ResNet-50 [20] from X-ray
images for classification of COVID-19 and non-COVID-19 cases. Ali et al. [21]
evaluated the performance of several existing deep models in diagnosing
COVID-19 in X-ray images. Islam et al. [22] on the other hand proposed a deep
framework combining CNNs and Recurrent Neural Networks (RNNs) for diagnosis of
COVID-19 in X-ray images. Initially, features are extracted with a CNN, which
are then feed into a Long short-term memory (LSTM) for diagnosis/detection
purposes. Kassani et al. [23] provide a detailed evaluation of several
existing deep models and classification algorithms to find a best combination
for COVID-19 diagnosis in both X-ray and CT scans. However, both modalities
are treated individually. The deep models are used for feature extraction,
which is then fed into different classification algorithms.

Some image-based COVID-19 diagnosis methods also rely on a recently introduced
concept of Federated Learning (FL) to ensure data privacy in a collaborative
learning environment, where several hospitals can participate in training a
global ML model. For instance, in [24] a deep model is collaboratively trained
in a federated learning environment on CT scans collected from different
sources. Kumar et al. [25] on the other hand proposed a blockchain-FL-based
framework for collecting data (CT scans) from different hospitals, and
collaboratively training a global deep model. Moreover, several exiting deep
models have also been evaluated in the proposed federated learning framework.
In [26], a federated learning technique is employed for training a global
model on electronic health records from various hospitals to predict mortality
within seven days in hospitalized COVID-19 patients.

###  II-B Challenges in Deploying ML at the Edge

####  II-B1 Resource Scarcity and Heterogeneity

Heterogeneous edge devices with varying computational, storage and
communication resources are a major bottleneck for the deployment of ML on the
edge. ML algorithms in general and deep learning (DL) in particular require a
large amount of computational and processing resources making the deployment
of ML impractical in several edge computing applications. DL models are
usually large and are computationally expensive, as both the training of a
deep model and its inferences are typically performed on power-hungry GPUs and
servers while, the edge devices are designed to be operated at low power and
usually have frugal memory, therefore, deploying DL models on the edge devices
is very challenging. One another important challenge is the availability of a
power source at edge device, i.e., a battery with long power backup is always
desirable in a typical edge computing network. In addition, the size of the
network and systems constraints is also a major challenge that can result in
only a few devices being active at a time [27].

####  II-B2 Network Communication

The heterogeneity of the computational and communication resources also led to
slow and unstable communication. In addition to resource heterogeneity, there
are other considerations as well, e.g., the Internet upload speed is typically
much slower than the download speed [2]. Therefore, in an edge computing
environment in which ML/DL models are being trained on the client site stable
and powerful Internet connection is always desirable, otherwise, the unstable
clients will be disconnected from the network that will result in a drop in
performance. On the other hand, deploying ML at the edge saves expensive
communication, i.e., we do not require the local (raw) data to be transmitted
to the cloud/server.

####  II-B3 Statistical Heterogeneity

The statistical heterogeneity, due to data generated by different types of
devices in an edge computing environment, can lead to many efficiency
challenges. For instance, the optimization/training of a ML/DL hyperparameters
becomes difficult, which directly affect its performance. To address the
statistical heterogeneity techniques such meta-learning can be used that can
enable device-specific modeling [28].

####  II-B4 Privacy and Security Challenges

Despite being able to train joint models with sharing data in a collaborative
learning environment using FL, privacy and security challenges arise with the
presence of malicious devices. For instance, an adversary can learn sensitive
information using the model parameters and the shared model. As shown in [29],
privacy-related information can be inferred from the shared weights even
without getting access to the data itself. To restrain leakage of privacy-
related information from the shared model, different privacy-preserving
techniques can be leveraged, such as cryptographic approaches and differential
privacy [30].

####  II-B5 Adversarial ML

Despite the state of the art performance of ML/DL techniques in solving
complex tasks, these techniques have been found vulnerable to carefully
crafted adversarial examples [31]. In a federated learning setup, a client or
multiple clients can be compromised to realize the attacks on the whole
network. For instance, local poisoning attacks using compromised attacker
devices are presented in [32]. The authors demonstrated that their proposed
attacks can increase the error rates of the distributively trained model on
four real-world datasets. Moreover, a systematic review focused on different
adversarial ML attacks and defenses for cloud-hosted ML models can be found in
[33].

###  II-C Enabling Technologies: Building Blocks for ML at Edge

####  II-C1 Schemes for deploying ML at the Edge

In recent years, enormous growth has been observed in the computational power
of edge devices, allowing them to play a more important role than just
collecting data in IoTs. ML can contribute significantly in fully utilizing
the potential of edge devices in numerous exciting applications (e.g., smart
healthcare using wearables technologies and AI-empowered sensors, etc.) and
turn them into more useful components of an IoT environment [34]. ML could be
employed at the edge in several ways, such as inference, sensor fusion,
transfer learning, generative models, and self-improving devices. In this
section, we briefly describe some of the most commonly used schemes.

  * •

Inference: The inference capabilities of ML, which aims predicting unseen
objects/classes based on the previous knowledge/trained data, help the IoTs to
perform different activities, such as cancer prognosis, brain tumor
classification, and other clinical data analysis at the edge devices resulting
in reduced latency and bandwidth in telemedicine [35].

  * •

Sensor Fusion: ML in conjunction with signal processing algorithms can be used
for the fusion of information from different sensors enabling efficient
utilization of the available information. With fusion capabilities, individual
sensors in an IoT environment can be converted into sophisticated synthetic
sensors to solve complex problems more accurately. For instance, in healthcare
data from several sensors/sources can be combined efficiently to predict a
clinical event, such as heart failure [36].

  * •

Transfer Learning: Transfer learning, which aims to re-utilize the knowledge
of one domain in another domain by fine-tuning a pre-trained model trained on
a larger dataset, can help them to learn on a smaller dataset with less
computational resources. In an IoT environment and in particular, in
healthcare applications where the data is scarcely available, the transfer
learning technique can be used to balance workload and latency where the pre-
trained models are put at the cloud and are shared among edge devices to be
fine-tuned for specific tasks [37].

  * •

Generative Models: Generative learning can also be useful in edge computing
where generative models can be used for the approximation of the original data
at the clouds to be used for training models at edge devices for applications
with less training samples or to solve complex tasks with minimal computation
from the clouds. Generative deep models have already been explored for the
generation of synthetic medical images [38].

  * •

Self-improving devices: In a typical IoT environment, ML techniques can also
be used to enable end devices to optimize their performance and improve
continuously based on the collected data and behaviors of other devices. Such
strategies help to configure the devices faster which ultimately leads to
faster and efficient implementation and deployment.

####  II-C2 Hardware Optimization Techniques

For successful deployment of ML at the edge, the two critical requirements of
edge computing—namely (i) low power consumption, and (ii) high
performance—need to be fulfilled. Thus, off-the-shelf solutions are not
practical to intelligent processing of data at the edge devices, and custom
hardware architectures need to be developed. In this section, we discuss some
hardware optimization techniques to optimize hardware resources for deploying
ML at the edge.

##### Decentrailized Distributed Computing

In edge computing, computations are completely or largely performed on end
devices in a distributed computing fashion. Also, edge computing brings data,
applications, and services closer to end devices while eliminating the need
for centralized clouds that requires infrastructure decentralization, such
kind of decentralization can be efficiently achieved using blockchain
technologies [39]. Therefore, computational resources can be shared among
end/edge devices by employing blockchain and smart contracts technologies thus
allowing computational resources demanding ML applications to be deployed at
the edge. For instance, different design requirements and challenges in
designing a decentralized edge computing and IoT ecosystem are presented in
[40]. This study is specifically focused on the need of using decentralized
trust schemes for the elimination of trust in centralized entities and
highlights the potential of using distributed ledger technology, i.e.,
blockchain for achieving the feature of decentralization. The backbone of
blockchain technologies is the distributed consensus mechanism enabling secure
communication among trust-less participants without the intervention of a
central controlling unit. There are many facets of blockchain with different
distributed consensus methods that can be used for edge-centric IoT systems
[41].

##### AI Co-Processors

Portable intelligent and dedicated co-processors are considered to be the
driving force for deploying AI/ML models at the edge. Different types of
specialized processors can be integrated into a single system or chip thus
forming a heterogeneous computing paradigm optimized for a specific type of
task. In general, AI co-processors have two common features: (1) enables
parallel computing using multiple mini-cores; (2) enables accelerated data
fetching using distributed memory that is placed right to mini-cores.

###  II-D Algorithmic Optimization Techniques

The development and advancement of ML algorithms are promising aspects that
facilitate the successful application of ML at the edge. In this regard,
various algorithms and techniques can be leveraged to enhance and reduce the
computation of the parameters in ML models by exploiting different properties
such as sparsity. The widely used methods are described below.

Figure 2: The proposed clustered federated learning based collaborative
learning paradigm (Fig. 2 (c)) versus the method of model training using
central data (Fig. 2 (a)) and the conventional federated learning model
training method for multiple modalities (Fig. 2 (b)). The term “clients”
refers to hospitals, clinics, and medical imaging facilities.

  * •

Parameter Efficient Networks: To efficiently deploy ML models at the edge,
computation and memory-efficient architectures of ML/DL models are highly
desirable. To facilitate embedded ML computing, various architectures of ML
models have been proposed in the literature that can be leveraged to deploy ML
models on the edge, e.g., Mobile Net [42] and SqueezeNet [43]. These
architectures are designed with a key focus on reducing computation costs
associated with the training and inferences of ML models while maintaining
accuracy.

  * •

Network Pruning: The literature suggests that a penalty of neurons in the
trained model does not contribute towards the final accuracy, therefore, such
neurons can be prune to save some memory. Google’s
Learn2Compress111https://ai.googleblog.com/2018/05/custom-on-device-ml-
models.html has found that neurons can be reduced by a factor of 2 while
retaining an overall accuracy of 97%. To this aim, several algorithms have
been proposed in the literature, such as learning important connections and
weights among neurons [44] and learning structural sparsity in deep models
[45]. Moreover, many ML models perform parameter computation using 32-bit
float values. On the other hand, edge devices typically operate on 8-bit
values or less. Therefore, the model size can be significantly reduced by
reducing precision.

  * •

Network Distillation: Network distillation is a method for transferring
knowledge learned by a larger model to a smaller model. Together with transfer
learning, which deals with the transfer of knowledge learned from one domain
to another domain, network distillation holds the substantial potential to
significantly reduce model size without comprising much on performances in
terms of accuracy. In addition, network distillation can be benefited from
other hyperparameters tuning algorithms as well. For instance, the
distillation method has been successfully used for application-specific and
resource-constrained IoT platforms [46].

##  III COVID-19 Diagnosis using Collaborative Federated Learning

In this section, we consider the problem of developing a single ML model for
classification of chest images from multiple sources (such as X-rays and
Ultrasound). Consider a clustered federated learning (CFL) setup as shown in
Figure 2 resembles the actual federated learning settings [47]. Clients in
each cluster represent the healthcare entities (remote medical imaging
facilities) and major hospitals or other government entities (e.g., ministry
of health) play the role of the cloud server facilitating the weights
aggregation and updates. The problem formulation for collaborative learning is
described below.

###  III-A Problem Formulation

In this task, we are interested in learning a shared model $M_{s}$ in a
collaborative fashion using clustered federated learning (CFL). As shown in
Figure 2, there are two clusters each having different kind of imaging
modality, i.e., cluster 1 ($C_{1}$) has clients having X-ray imagining
facility and clients in cluster 2 ($C_{2}$) posses ultrasound imagining
facilities, therefore, each cluster $C_{k}$ is disjoint and has different data
distribution $\mathcal{D}_{k}$. Each client $m$ in cluster $C_{k}$ has drawn
its samples $z^{k,1},\dots,z^{k,m}$ from the distribution $\mathcal{D}_{k}$
such that there are no overlapping samples among the clients. We have
formulated the problem of collaborative learning as supervised learning
problem such that each sample $z^{k,m}$ contains a pair of data sample
$x^{k,m}$ and its corresponding class label $y^{k,m}$, denoted by
$z^{k,m}=(x^{k,m},y^{k,m})$. Furthermore, we assume that each client does not
have any knowledge either about the identity and data of every other client
within the same cluster as well as in the other cluster. The major hospital
(aka server) shares a shared model $M_{s}$ and initial weights $W_{0}$ with
each client of every cluster. After receiving the $M_{s}$ and $W_{0}$, each
client trains the shared model (i.e., $M_{s}$) using its own local data
$D_{k,m}$, where $k=\\{1,2\\}$ and $m$ denotes the number of clients in each
cluster $C_{k}$. After that, every client in each cluster shares the learned
weights $W_{r,k,m}$ to the server, where $r$ denotes the communication
round/iteration number. After receiving the weight updates from each client,
the server performs federated averaging using Eq. 1.

$W_{r}=\frac{1}{n}\sum_{i}^{n}w_{i}\times W_{i}$ (1)

Where, $n$ denotes the total number of clients participating in the CFL setup
(i.e., $n=|C_{1}|+|C_{2}|$) and $w$ is a weighting factor that specifies the
weight-age given to the weights of each client. Then the server updates the
new weights (i.e., update its copy of $M_{s}$ with $W_{r}$) and performs the
inference using its multi-modal test data (the two modalities, i.e., X-ray and
Ultrasound are merged to make the testing data multi-modal). After testing the
performance of $M_{s}$ at the communication round $r$, the server shares the
updated weights $W_{r}$ with all clients in each cluster and repeats the
process until the specified criteria or desired performance is achieved. The
algorithm for collaborative multi-modal learning using CFL is presented in
Algorithm 1.

Input: Shared Model $M_{s}$, Clusters $k$, Initial Model Weights $W_{0}$, Set
of Clients $m$, Communication Rounds $R$, Epochs $E$, Batch Size $B$, and
Learning Rate $\eta$

Output: Updated Weights $W_{r}$

Initialize: $W_{0}$, $R$, $E$, $B$, and $\eta$ for  _$r=1,...,R$_ do

for  _$i=1,...,k$ in parallel_ do

for  _$j=1,...,m$ in parallel_ do

if  _$r==1$_ then

$W_{i,m}\leftarrow W_{0}$;

$M_{s}\leftarrow W_{0}$;

for  _$e\in E$_ do

Using $B$ & $\eta$;

Train $M_{s}$ using $z^{i,j}=(x^{i,j},y^{i,j})$;

Get $W_{j}$ from $M_{s}$;

end for

else

$M_{s}\leftarrow W_{r}$;

for  _$e\in E$_ do

Using $B$ & $\eta$;

Train $M_{s}$ using $z^{i,j}=(x^{i,j},y^{i,j})$;

Get $W_{j}$ from $M_{s}$;

end for

end if

end for

$W_{i,r}=\frac{1}{m}\sum_{j=1}^{m}w_{j}\times W_{j}$;

end for

Return: $W_{r}=\frac{1}{k}\sum_{i=1}^{k}W_{i,r}$;

end for

Algorithm 1 Collaborative learning from data of different sources and modality
using CFL

###  III-B Experimental Setup

####  III-B1 Data Description

For this study, two datasets from different sources one containing chest X-ray
[10] and chest ultrasound images [13], are used. We formulated the problem as
binary classification, i.e., differentiating between COVID-19 chest images and
normal chest images. Each dataset is divided into two parts, i.e., a training
set and a testing set using a split of 80% and 20%, respectively. The training
portion (i.e., 80%) of each dataset is further divided into different parts,
depending upon the number of clients in that cluster. The distribution of
training and testing data of X-ray and Ultrasound datasets over different
classes is shown in Table I. Moreover, the testing sets from both datasets are
merged to develop a joint testing set that will be used by the server for the
evaluation of the performance of a shared model that is being trained in a
collaborative fashion using CFL.

We further note that the datasets used in this study have inter and intra
class variability in terms of image size and quality, contrast and brightness
level, and positioning of subjects, an example is shown in Figure 3. This is
not surprising as these publicly available databases are not standard datasets
for COVID-19 detection, and have been curated from different sources [48].
Moreover, it is evident from Table I that these datasets are highly
imbalanced. These limitations make the training of a generalized model more
difficult.

Figure 3: The depiction of inter and intra class variations observed in COVID-19 datasets (X-ray [10] and Ultrasound [13]). TABLE I: The distribution of training and testing data of X-ray and Ultrasound datasets over different classes. Data | Class | Training Data (80%) | Test Data (20%)
---|---|---|---
X-ray | COVID-19 | 179 | 44
Healthy | 1072 | 269
Ultrasound | COVID-19 | 319 | 80
Healthy | 116 | 30

####  III-B2 Model Architecture and Implementation Details

In our experiments, we have used the VGG16 model with one extra convolutional
layer and three fully connected layers stacked before its original output
layer. Each image is first converted into a gray-scale image, which is then
resized to a dimension of $256\times 256$. Moreover, the resized images are
normalized before feeding into the model. The model is trained using Adam
optimizer with a learning rate of $0.0001$ at each client. We use different
types of standard data augmentation techniques for training the models.
Furthermore, to address the problem of imbalanced classes, we propose to use
focal loss [49], which is suited for such issues in binary classification
tasks. The focal loss adds a modulating factor $(1-p_{t})^{\gamma}$ to the
standard cross-entropy loss, where $\gamma\geq 0$ is a tunable focusing
parameter. The $\alpha$-balanced variant of focal loss is defined in Eq. 2,
where $\alpha$ balances the importance of positive/negative examples [49].

$FL(p_{t})=-\alpha_{t}(1-p_{t})^{\gamma}\log(p_{t})$ (2)

We note that for the implementation of the proposed work we used TensorFlow ML
library, and all experiments are performed in a simulated environment. The
results of the different experiments are described in the next section.

##  IV Results and Discussions

In order to show the effectiveness of the proposed multi-modal collaborative
learning framework for COVID-19 diagnosis, we performed several experiments.
On one side, we aim to evaluate and compare the performances of CFL against
two baselines, namely (i) specialized FL baseline, and the (ii) multi-
modal222By the term multi-modal we mean images acquired using different
imagining techniques, i.e., modalities (e.g., X-ray and Ultrasound).
conventional FL. Since CFL aims to tackle the convergence issues of
conventional FL schemes due to the diverse distribution of the data, the two
baselines, we believe, are appropriate options as a comparison benchmark
instead of the state of the art methods for COVID-19 diagnosis. We note that
due to the limitations of the dataset, we only consider the divergence in
distribution of the data in terms of the nature of the data (i.e., the
distribution of ultrasound and X-ray images is different). The first baseline
shows the best-case scenario, where separated models for each type of imagery,
which we termed as specialized models, are trained in a FL environment. The
individual models are trained on X-ray and Ultrasound images with a learning
rate of $0.0001$ and a batch size of $32$ resulting into two separate models
one for each modality (i.e., X-ray and Ultrasound). The second baseline
represents the experimental setup of a conventional FL environment, where the
data is distributed among different clients, and a shared ML model is built in
a federated environment. The parameters used in different experiments can be
found in Table II.

TABLE II: Parameters of clustered federated learning (CFL) experiments. Parameter (s) | Value (s)
---|---
Communication Rounds | 30, 50, & 100
Epochs | 5 & 10
Batch Size | 16 & 32
Learning Rate | $1e^{-3}$
TABLE III: COMPARISON AGAINST THE TWO BASELINES IN TERMS OF PRECISION, RECALL, AND F1-SCORE. Promising results are obtained by CFL, outperforming the conventional FL while slightly lower performance is obtained compared to Central Baseline with the added advantage of improved privacy and data security. ∗A separate model is trained in federated learning settings for each modality. Dataset | Class | Federated Learning (Specialized∗) | Federated Learning (multi-modal) | Clustered FL (multi-modal)
---|---|---|---|---
Precision | Recall | F1-Score | Precision | Recall | F1-Score | Precision | Recall | F1-Score
X-ray | COVID-19 | 0.73 | 0.82 | 0.77 | 0.30 | 0.68 | 0.41 | 0.71 | 0.82 | 0.76
Healthy | 0.97 | 0.95 | 0.96 | 0.93 | 0.74 | 0.82 | 0.97 | 0.94 | 0.96
Ultrasound | COVID-19 | 0.97 | 0.95 | 0.97 | 0.94 | 0.76 | 0.84 | 0.93 | 0.95 | 0.94
Healthy | 0.88 | 0.93 | 0.90 | 0.58 | 0.87 | 0.69 | 0.86 | 0.80 | 0.83

(a) X-ray

(b) Ultrasound

Figure 4: Comparison of clustered federated learning (CFL) with two baselines
(i.e., the specialized models (trained with conventional FL independently for
each modality) and conventional federated learning (when the model is trained
using multi-modal data)) in terms of average values of precision, recall, and
F1-score on X-ray and Ultrasound imagery.

Table III and Figure 4 provides the experimental results per class and overall
(per dataset) results, respectively, in terms of precision, recall, and
F1-Score. Since the data set is not balanced, so we believe, alone accuracy is
not enough to evaluate the proposed method. For performance evaluation of the
three experimental setups (i.e., the two baselines and CFL), we kept the
similar experimental setup where we first train the baseline models with a
batch size of 16 (for each modality) and then we train the same model in CFL
fashion (i.e., using multi-modal settings) with 5 epochs of local training
with a batch size of 16. Then we evaluated the collaboratively trained model
with the test data from each cluster (modality), i.e., X-ray and Ultrasound.
As can be seen in the Figure 4, overall comparable results are observed for
multi-modal model trained using CFL compared with the specialized two models
trained in a conventional FL environment using X-ray and Ultrasound imagery
separately. On the other hand, we can see that CFL performance is considerably
better than the performance of multi-modal model trained in a conventional
federated learning environment. Moreover, it is evident from the figure that a
collaboratively trained model is capable of recognizing the test of images
from different modalities without having explicit knowledge about these
modalities. Moreover, overall better results are obtained on ultrasound images
(Figure 4(b)) compared to X-ray imagery (Figure 4(a)) for all models.

Figure 5: Comparison of clustered federated learning (CFL) with the
specialized models (trained with conventional FL independently for each
modality, i.e., X-ray and Ultrasound) and conventional federated learning
(when the model is trained using multi-modal data) over increasing number of
communication rounds.

In Figure 5, we provide the comparison of the three experimental setups (i.e.,
specialized models trained in conventional FL settings, multi-modal models
trained in a conventional FL and CFL environments) in terms of accuracy and
loss at different communication rounds. The figure depicts that the proposed
CFL model (which is trained using multi-modal data) provides comparable
performance with that of specialized FL models (that are separately trained
for each modality). Moreover, it is also evident from the figure that the
model trained using multi-modal data in conventional FL settings gets over-
fitted after 50 communication rounds. On the counter side, the model keeps on
learning in CFL setting, though it also tends to show over-fitting behavior at
later stage communication round as evident in the Figure 5. The vertical red
line shown on Figures 5 and 5 shows the inflection point beyond which the
parameters of the specialized machine learning models of the two clusters
(i.e., X-ray and ultrasound) start to diverge from each other. This diversion
limits the extent to which the multimodal model can be generalized to fit the
underlying multimodal data (i.e., X-ray and ultrasound). Therefore, Figure 5
provides the insight that the federated learning rounds should be stopped as
soon as the inflection point in the value of the loss function is reached.
This inflection point identified the number of rounds beyond which the
multimodal machine learning model cannot be enhanced.

###  IV-A Lessons Learned

Some key lessons learned from the literature and experiments conducted in this
work are:

  * •

CFL ensures the privacy of the user’s local data, as it does not need to be
shared with the server for central training.

  * •

The communication payload of model weights is far less than the payload of
sharing actual data, therefore, it saves bandwidth and as well as time.

  * •

It enables the collaborative learning of multi-modal features by shared model
$M_{s}$ without sharing any explicit information about the modality of the
local data and the data itself.

  * •

More importantly, compared to the conventional FL, CFL ensures better
performance in presence of divergence in the data distribution. The divergence
in the distribution could be in terms of the distribution of negative and
positive samples per class as well as in terms of the nature of data samples
as detailed earlier.

  * •

This particular use-case demonstrates the potential of the method in medical
applications where remote smaller healthcare units can benefit from this
collaborative learning method.

However, despite these benefits, there are some challenges and limitations as
well, e.g., efficiency, security issues, and the optimization of CFL
parameters is difficult. Moreover, there is a trade-off in the model
performance when we compare model trained using central data and model trained
with distributed data using federate learning. In addition, for multi-modal
distributively dispersed data, the development of personalized models that are
tailored to these modalities is required for local training, which will
enhance the efficiency of the shared model and as well as of the models on the
client-side. For instance, from our experiments we have learned that the
performance of the model being trained in CFL settings start degrading after a
particular point (i.e., communication round). Thus highlighting the need of
early stopping and the development of optimal stopping criteria except the
maximum allowed communication rounds.

##  V Open Research Issues

###  V-A Developing Personalized Approaches

The edge computing network is potentially more heterogeneous as compared to
any other central network and clients in an edge computing network vary due to
data acquisition resources, network, computational, and storage resources,
etc. Moreover, as discussed above in the paper, clients can significantly vary
due to statistical heterogeneity, which is usually a great challenge in
realistic settings. For example, as we discussed in the above section,
developing a multi-modal collaborative learning framework for COVID-19
diagnosis has efficiency challenges due to the aforementioned heterogeneity
issues. Therefore, to handle such heterogeneities, the development of
personalized and client-specific ML/DL approaches is required.

###  V-B Adversarialy Robust ML

The edge computing network is more prone to security threats, as edge
computing network is an ideal environment for adversaries that aim to get
desired outcomes or incentives for breaching the network security and privacy
of participating agents. This phenomenon becomes, even more, worse with the
integration of ML/DL models that are vulnerable to adversarial attacks, which
have been already shown effective for healthcare applications [50]. For
instance, an adversarial attack on CT scanners in an actual hospital
environment by manipulating the hospital’s network has already been realized
in the literature [51] and threats of adversarial ML for ML and IoT empowered
COVID-19 detection systems are highlighted in [52]. To restrain the
adversarial attacks, different defensive techniques have been proposed in the
literature. However, the adversarially robust methods developed so far are
attack specific, i.e., they only work for particular attacks for which they
were developed and fail to withstand unforeseen attacks. Therefore, the
development of adversarially robust ML/DL models is still an open research
problem that demands a proportionate amount of interest from the community
with the advancement of ML/DL techniques. Moreover, for the successful
deployment of ML/DL models on the edge, in particular, for developing robust
healthcare applications, the development of adversarially robust models are of
utmost importance.

###  V-C Asynchronous Distributed ML

In distributed computing, two approaches are widely used for communication,
i.e., synchronous and asynchronous. These approaches are ideal for scenarios
where data is instantly available for instance in the central picture
archiving and communication system (PACS) of a hospital. However, in realistic
settings, the data collection or acquisition might get delayed due to any
reason, such as due to some network issue or unavailability of I/O device,
etc. Moreover, it is possible that the client (i.e., a small healthcare
entity) in an ML-based collaborative computing network is not active at the
current iteration/communication round due to some inherent issue, this will
result in a delay in the federated parameters update process and will
eventually affect the system’s overall performance. Therefore, it is worth
studying and developing asynchronous approaches for facilitating shared model
training for healthcare applications using distributed data.

##  VI Conclusions

This article provides insights on how edge computing and machine learning
advances can be used to provide a solution for COVID-19 diagnosis in an
efficient privacy-aware manner thereby allowing remote healthcare units to
benefit from collaborative learning paradigm without sharing local data. In
particular, we propose using a clustered federated learning (CFL)-based
collaborative learning framework to intelligently process visual data at the
edge by training a multi-modal ML model capable of diagnoses COVID-19 in both
X-ray and Ultrasound imagery. Compared to the conventional FL, CFL is found to
better cope with the divergence in distribution of data from different sources
(i.e., X-ray and Ultrasound imagery). In the current implementation, we
consider the divergence in distribution due to the sources and nature of the
data due to the limitations of the datasets. In the future, we will explore
how CFL performs in the presence of variances in the distribution of the data
in terms of the number of samples per client.

## References

  * [1] Kashif Ahmad, Majdi Maabreh, Mohamed Ghaly, Khalil Khan, Junaid Qadir, and Ala Al-Fuqaha.  Developing future human-centered smart cities: Critical analysis of smart city security, interpretability, and ethical challenges.  arXiv preprint arXiv:2012.09110, 2020.
  * [2] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao.  Federated learning in mobile edge networks: A comprehensive survey.  IEEE Communications Surveys & Tutorials, 2020.
  * [3] S. Latif, M. Usman, S. Manzoor, W. Iqbal, J. Qadir, G. Tyson, I. Castro, A. Razi, M. N. K. Boulos, A. Weller, and J. Crowcroft.  Leveraging data science to combat covid-19: A comprehensive review.  IEEE Transactions on Artificial Intelligence, 1(1):85–103, 2020\.
  * [4] Yi-Wei Tang, Jonathan E Schmitz, David H Persing, and Charles W Stratton.  Laboratory diagnosis of covid-19: current issues and challenges.  Journal of clinical microbiology, 58(6), 2020.
  * [5] Mohamed A Hendaus.  Remdesivir in the treatment of coronavirus disease 2019 (covid-19): A simplified summary.  Journal of Biomolecular Structure and Dynamics, 2020.
  * [6] Peter J Hotez, David B Corry, and Maria Elena Bottazzi.  COVID-19 vaccine design: the Janus face of immune enhancement.  Nature Reviews Immunology, 20(6):347–348, 2020.
  * [7] Tao Ai, Zhenlu Yang, Hongyan Hou, Chenao Zhan, Chong Chen, Wenzhi Lv, Qian Tao, Ziyong Sun, and Liming Xia.  Correlation of chest ct and rt-pcr testing in coronavirus disease 2019 (covid-19) in china: a report of 1014 cases.  Radiology, page 200642, 2020.
  * [8] Roman Kalkreuth and Paul Kaufmann.  COVID-19: a survey on public medical imaging data resources.  arXiv preprint arXiv:2004.04569, 2020.
  * [9] Halgurd S Maghdid, Aras T Asaad, Kayhan Zrar Ghafoor, Ali Safaa Sadiq, and Muhammad Khurram Khan.  Diagnosing COVID-19 pneumonia from X-ray and CT images using deep learning and transfer learning algorithms.  arXiv preprint arXiv:2004.00038, 2020.
  * [10] Joseph Paul Cohen, Paul Morrison, Lan Dao, Karsten Roth, Tim Q Duong, and Marzyeh Ghassemi.  COVID-19 Image Data Collection: Prospective Predictions Are the Future.  arXiv 2006.11988, 2020.
  * [11] COVID-19 CT segmentation dataset.  http://medicalsegmentation.com/covid19/.  Accessed: 2020-08-127.
  * [12] Jinyu Zhao, Yichen Zhang, Xuehai He, and Pengtao Xie.  Covid-ct-dataset: a ct scan dataset about covid-19.  arXiv preprint arXiv:2003.13865, 2020.
  * [13] Jannis Born, Gabriel Brändle, Manuel Cossio, Marion Disdier, Julie Goulet, Jérémie Roulin, and Nina Wiedemann.  POCOVID-Net: automatic detection of COVID-19 from a new lung ultrasound imaging dataset (POCUS).  arXiv preprint arXiv:2004.12084, 2020.
  * [14] Shuai Wang, Bo Kang, Jinlu Ma, Xianjun Zeng, Mingming Xiao, Jia Guo, Mengjiao Cai, Jingyi Yang, Yaodong Li, Xiangfei Meng, et al.  A deep learning algorithm using ct images to screen for corona virus disease (covid-19).  MedRxiv, 2020.
  * [15] Charmaine Butt, Jagpal Gill, David Chun, and Benson A Babu.  Deep learning system to screen coronavirus disease 2019 pneumonia.  Applied Intelligence, page 1, 2020.
  * [16] Lin Li, Lixin Qin, Zeguo Xu, Youbing Yin, Xin Wang, Bin Kong, Junjie Bai, Yi Lu, Zhenghan Fang, Qi Song, et al.  Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT.  Radiology, 2020.
  * [17] Parnian Afshar, Shahin Heidarian, Farnoosh Naderkhani, Anastasia Oikonomou, Konstantinos N Plataniotis, and Arash Mohammadi.  COVID-CAPS: A capsule network-based framework for identification of COVID-19 cases from X-ray images.  arXiv preprint arXiv:2004.02696, 2020.
  * [18] Linda Wang and Alexander Wong.  Covid-net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-Ray images.  arXiv preprint arXiv:2003.09871, 2020.
  * [19] Prabira Kumar Sethy and Santi Kumari Behera.  Detection of coronavirus disease (covid-19) based on deep features.  Preprints, 2020030300:2020, 2020.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [21] Ali Narin, Ceren Kaya, and Ziynet Pamuk.  Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks.  arXiv preprint arXiv:2003.10849, 2020.
  * [22] Md Zabirul Islam, Md Milon Islam, and Amanullah Asraf.  A combined deep cnn-lstm network for the detection of novel coronavirus (covid-19) using x-ray images.  Informatics in Medicine Unlocked, page 100412, 2020.
  * [23] Sara Hosseinzadeh Kassani, Peyman Hosseinzadeh Kassasni, Michal J Wesolowski, Kevin A Schneider, and Ralph Deters.  Automatic detection of coronavirus disease (covid-19) in x-ray and ct images: A machine learning-based approach.  arXiv preprint arXiv:2004.10641, 2020.
  * [24] Yongchao Xu, Liya Ma, Fan Yang, Yanyan Chen, Ke Ma, Jiehua Yang, Xian Yang, Yaobing Chen, Chang Shu, Ziwei Fan, et al.  A collaborative online AI engine for CT-based COVID-19 diagnosis.  medRxiv, 2020.
  * [25] Rajesh Kumar, Abdullah Aman Khan, Sinmin Zhang, WenYong Wang, Yousif Abuidris, Waqas Amin, and Jay Kumar.  Blockchain-federated-learning and deep learning models for COVID-19 detection using CT imaging.  arXiv preprint arXiv:2007.06537, 2020.
  * [26] Akhil Vaid, Suraj K Jaladanki, Jie Xu, Shelly Teng, Arvind Kumar, Samuel Lee, Sulaiman Somani, Ishan Paranjpe, Jessica K De Freitas, Tingyi Wanyan, et al.  Federated learning of electronic health records improves mortality prediction in patients hospitalized with covid-19.  medRxiv, 2020.
  * [27] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith.  Federated learning: Challenges, methods, and future directions.  IEEE Signal Processing Magazine, 37(3):50–60, 2020.
  * [28] Jeffrey Li, Mikhail Khodak, Sebastian Caldas, and Ameet Talwalkar.  Differentially private meta-learning.  arXiv preprint arXiv:1909.05830, 2019.
  * [29] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov.  Exploiting unintended feature leakage in collaborative learning.  In 2019 IEEE Symposium on Security and Privacy (SP), pages 691–706. IEEE, 2019.
  * [30] Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala Al-Fuqaha.  Secure and robust machine learning for healthcare: A survey.  IEEE Reviews in Biomedical Engineering, 2020.
  * [31] Adnan Qayyum, Muhammad Usama, Junaid Qadir, and Ala Al-Fuqaha.  Securing connected & autonomous vehicles: Challenges posed by adversarial machine learning and the way forward.  IEEE Communications Surveys & Tutorials, 22(2):998–1026, 2020\.
  * [32] Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong.  Local model poisoning attacks to byzantine-robust federated learning.  arXiv preprint arXiv:1911.11815, 2019.
  * [33] Adnan Qayyum, Aneeqa Ijaz, Muhammad Usama, Waleed Iqbal, Junaid Qadir, Yehia Elkhatib, and Ala Al-Fuqaha.  Securing machine learning in the cloud: A systematic review of cloud machine learning security.  Frontiers in Big Data, 3:43, 2020.
  * [34] Guangxu Zhu, Dongzhu Liu, Yuqing Du, Changsheng You, Jun Zhang, and Kaibin Huang.  Toward an intelligent edge: wireless communication meets machine learning.  IEEE Communications Magazine, 58(1):19–25, 2020.
  * [35] Graham Gobieski, Brandon Lucia, and Nathan Beckmann.  Intelligence beyond the edge: Inference on intermittent embedded systems.  In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 199–213. ACM, 2019.
  * [36] Chirag Nagpal.  Deep multimodal fusion of health records and notes for multitask clinical event prediction.  31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA., 2017.
  * [37] Zhenyu Zhou, Haijun Liao, Bo Gu, Kazi Mohammed Saidul Huq, Shahid Mumtaz, and Jonathan Rodriguez.  Robust mobile crowd sensing: When deep learning meets edge computing.  IEEE Network, 32(4):54–60, 2018.
  * [38] Adnan Qayyum, Waqas Sultani, Fahad Shamshad, Junaid Qadir, and Rashid Tufail.  Single-shot retinal image enhancement using deep image priors.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 636–646. Springer, 2020.
  * [39] David A Booz, Jonathan D Dye, Michael J Dye, and Egan F Ford.  Decentralized autonomous edge compute coordinated by smart contract on a blockchain, September 28 2017.  US Patent App. 15/082,559.
  * [40] Ioannis Psaras.  Decentralised edge-computing and IoT through distributed trust.  In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pages 505–507. ACM, 2018.
  * [41] Yang Zhao, Jun Zhao, Linshan Jiang, Rui Tan, Dusit Niyato, Zengxiang Li, Lingjuan Lyu, and Yingbo Liu.  Privacy-preserving blockchain-based federated learning for iot devices.  IEEE Internet of Things Journal, 2020.
  * [42] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [43] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and¡ 0.5 MB model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [44] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  In Advances in neural information processing systems, pages 1135–1143, 2015.
  * [45] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li.  Learning structured sparsity in deep neural networks.  In Advances in neural information processing systems, pages 2074–2082, 2016.
  * [46] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133–4141, 2017.
  * [47] Felix Sattler, Klaus-Robert Müller, and Wojciech Samek.  Clustered federated learning: Model-agnostic distributed multi-task optimization under privacy constraints.  arXiv preprint arXiv:1910.01991, 2019.
  * [48] Michael J Horry, Subrata Chakraborty, Manoranjan Paul, Anwaar Ulhaq, Biswajeet Pradhan, Manas Saha, and Nagesh Shukla.  COVID-19 detection through transfer learning using multimodal imaging data.  IEEE Access, 8:149808–149824, 2020.
  * [49] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [50] Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane.  Adversarial attacks on medical machine learning.  Science, 363(6433):1287–1289, 2019.
  * [51] Yisroel Mirsky, Tom Mahler, Ilan Shelef, and Yuval Elovici.  Ct-gan: Malicious tampering of 3d medical imagery using deep learning.  In 28th USENIX Security Symposium USENIX Security 19), pages 461–478, 2019.
  * [52] Abdur Rahman, M Shamim Hossain, Nabil A Alrajeh, and Fawaz Alsolami.  Adversarial examples–security threats to COVID-19 deep learning systems in medical IoT devices.  IEEE Internet of Things Journal, 2020.